/tf)  £<?<? 


ESD-TR-69-302 

Jf-STC  FjCLSB  Co  P[ 


-oO  ACCESSION  LIST 

ESTJ  Call  Nfc_  0/689 

Copy  No. 


SYSTEMS  MANAGEMENT  APPLIED  TO  LARGE  COMPUTER 
PROGRAMS  IN  BUIC  III;  REVIEW  OF  EXPERIENCE 


Lloyd  V.  Searle 
Perry  E.  Rosove 
Eugene  H.  Sydow 


June  1969 


ESO  RECORD  COPY 

return  to 

SCIENTIFIC  l  TECHACM 

lESIli  BONJWWC  mi 


AIR  WEAPONS  SURVEILLANCE  AND  CONTROL  SPO 

ELECTRONIC  SYSTEMS  DIVISION 

AIR  FORCE  SYSTEMS  COMMAND 

UNITED  STATES  AIR  FORCE 

L.  G.  Hanscom  Field,  Bedford,  Massachusetts 


This  document  has  been 
approved  for  public  release  and 
sale;  its  distribution  is 
unlimited. 


(Prepared  under  Contract  No.  FI9628-67-C-0026  by  System  Development 
Corporation,  2500  Colorado  Avenue,  Santa  Monica,  California  90406.) 


ADi)M$r 


LEGAL  NOTICE 


When  U.  S.  Government  drawings,  specifications  or  other  data  are  used  for  any 
purpose  other  than  a  definitely  related  government  procurement  operation,  the 
government  thereby  incurs  no  responsibility  nor  any  obligation  whatsoever;  and 
the  fact  that  the  government  may  have  formulated,  furnished,  or  in  any  way  sup¬ 
plied  the  said  drawings,  specifications,  or  other  data  is  not  to  be  regarded  by 
implication  or  otherwise  as  in  any  manner  licensing  the  holder  or  any  other  person 
or  conveying  any  rights  or  permission  to  manufacture,  use,  or  sell  any  patented 
invention  that  may  in  any  way  be  related  thereto. 


OTHER  NOTICES 


Do  not  return  this  copy.  Retain  or  destroy. 


ESD-TR-69-302 


SYSTEMS  MANAGEMENT  APPLIED  TO  LARGE  COMPUTER 
PROGRAMS  IN  BUIC  III;  REVIEW  OF  EXPERIENCE 


Lloyd  V.  Searle 
Perry  E.  Rosove 
Eugene  H.  Sydow 


June  1969 


AIR  WEAPONS  SURVEILLANCE  AND  CONTROL  SPO 

ELECTRONIC  SYSTEMS  DIVISION 

AIR  FORCE  SYSTEMS  COMMAND 

UNITED  STATES  AIR  FORCE 

L.  G.  Hanscom  Field,  Bedford,  Massachusetts 


This  document  has  been 
approved  for  public  release  and 
sale;  its  distribution  is 
unlimited.  _ 


(Prepared  under  Contract  No.  FI9628-67-C-0026  by  System  Development 
Corporation,  2500  Colorado  Avenue,  Santa  Monica,  California  90406.) 


1 


FOREWORD 


This  report  is  the  result  of  a  special  study  which  was  carried  out  at  the 
System  Development  Corporation  offices  in  Santa  Monica,  California.  The  study 
was  a  portion  of  the  contract  work  for  the  BUIC  III  System  Program  during  the 
period  of  July  1968  to  June  1969.  The  purpose  of  the  study  was  to  provide  a 
documentary  record  of  the  contractor’s  experiences  in  applying  Air  Force  systems 
management  techniques  to  the  BUIC  III  computer  program  acquisition. 

Material  contained  in  this  report  represents  a  revision  of  material  ori¬ 
ginally  reported  in  the  SDC  Technical  Memorandum  TM-4223,  “Systems  Management 
Applied  to  Information  Processing  Elements  in  BUIC  III;  Review  of  Experience," 
dated  25  February  1969.  The  content  is  based  on  information  which  was  collected, 
organized,  and  interpreted  by  the  contractor;  it  does  not  reflect  official 
information  provided  by  the  416M  System  Program  Office  (SPO)  nor  technical 
participation  by  SPO  personnel. 

Administrative  monitoring  of  the  task  was  accomplished  by  Major  Harvey  B. 
Blanton  and  Captain  James  E.  Puffer,  Office  Symbol  ESSGB,  of  the  Air  Weapons 
Surveillance  and  Control  System  Program  Office  (416M/P/418L) ,  Electronic 
Systems  Division,  Air  Force  Systems  Command,  Laurence  G.  Hanscom  Field, 

Bedford,  Massachusetts . 

Publication  of  this  report  does  not  constitute  Air  Force  approval  of  the 
report’s  findings  or  conclusions.  It  is  published  only  for  the  exchange  and 
stimulation  of  ideas. 


LIONEL  C.  ALLARD,  JR.,  Colonel  USAF 
System  Program  Director 


Air  Weapons  Surveillance  and  Control 
System  Program  Office  (4i6m/p/4i8l) 
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ABSTRACT 

This  report  is  a  review  and  analysis  of  experience  with  the 
application  of  Air  Force  systems  management  techniques  to  the 
acquisition  of  information  processing  elements  in  the  416M 
(BUIC  III)  system  program.  The  report  includes  a  background 
review  of  the  systems  management  concepts  and  trends  in  relation 
to  practices  which  had  been  employed  in  L-system  programs  pre¬ 
ceding  BUIC  III.  Novel  requirements  introduced  in  BUIC  III 
are  identified  in  the  areas  of  computer  program  configuration 
management,  standard  documentation,  design  reviews,  and  Category 
I  testing;  and  a  summary  is  presented  of  the  milestones  associated 
with  these  requirements  as  they  actually  occurred  during  the 
program.  Finally,  the  experience  in  specific  areas  is  discussed 
and  evaluated  with  respect  to  implications  for  future  modification 
and  use  of  the  management  techniques. 
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CHAPTER  I 


INTRODUCTION 


A.  SCOPE  AND  OBJECTIVES 

This  report  is  concerned  with  the  problem  of  the  integration  of  a  new  type  of 
technology  and  the  management  process  in  the  military  realm.  The  technology 
is  that  of  computer-based  command  and  control  systems.  The  particular  focus 
of  this  report  is  upon  the  computer  programs  and  associated  products.  Manage¬ 
ment,  in  this  context,  refers  to  the  process  whereby  electronic  command  and 
control  systems  are  acquired  by  a  procurement  agency  for  a  using  command.  The 
system  on  which  we  shall  focus  is  known  as  BUIC  III  (416M) ,  the  back-up  inter¬ 
ceptor  control  system  for  the  SAGE  (416L)  system  of  air  defense. 

The  procurement  agency  in  this  example  of  the  system  acquisition  process  is 
the  Electronic  Systems  Division  (ESD)  of  the  Air  Force  Systems  Command  (AFSC) 
while  the  Air*  Defense  Command  (ADC)  is  the  user.  Two  non-military,  not-for- 
profit  organizations  of  critical  concern  in  this  report  are  the  MITRE  Corpora¬ 
tion,  which  was  the  General  Systems  Engineering/Technical  Direction  Contractor 
(GSE/TDC)  in  BUIC  III,  and  the  System  Development  Corporation  (SDC) ,  which  was 
responsible  for  the  BUIC  III  operational  computer  program  system  segment. 

Other  non-military  organizations,  of  course,  also  played  important  roles  in 
the  development  of  BUIC  III.  Foremost  among  these  is  the  Burroughs  Corporation, 
which  designed  and  manufactured  the  system  equipment.  However,  since  the  focus 
of  attention  in  this  report  is  on  the  acquisition  of  computer  programs,  the 
roles  of  these  other  organizations  will  be  dealt  with  only  insofar  as  they 
impinge  upon  the  main  areas  of  concern. 

In  the  BUIC  III  system  project,  several  management  techniques  based  on  the 
AFSCM  375-series  and  associated  systems  management  principles  were  applied  on 
a  broad  scale  for  the  first  time  to  the  acquisition  of  information  processing 
elements  of  a  command  and  control  system.  While  some  of  the  techniques  have 
now  been  adopted  as  formal  Air  Force  requirements  for  general  use,  BUIC  III 
represents  a  first  and  significant  case  of  actual  experience  with  their 
systematic  application  in  a  major  system  project.  Because  of  the  increasing 
importance  of  information  processing  elements  in  Air  Force  systems,  that 
experience  constitutes  an  invaluable  source  of  potential  guidance  for  future 
applications,  as  well  as  for  continued  expansion,  modification,  and  refinement 
of  the  techniques.  It  is  therefore  a  primary  objective  of  this  report  to 
document  the  history  of  the  program  in  such  a  way  as  to  make  the  lessons 
learned  available  for  early  dissemination  and  use. 


*  Now  Aerospace  Defense  Command. 
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B.  METHODS 

Two  main  types  of  sources  were  used  to  gather  the  information  upon  which  this 
report  is  based:  (1)  documents  of  various  types,  and  (2)  interviews  with  SDC 
personnel  who  participated  in  the  BUIC  III  system  program. 

The  documents  used  to  reconstruct  the  history  of  the  BUIC  III  acquisition 
effort  and  the  problems  encountered  by  SDC  during  the  course  of  that  effort 
were  derived  primarily  from  SDC  sources.  These  included:  historical  reports 
on  SAGE,  BUIC  II,  and  BUIC  III  written  by  SDC  personnel;  letters,  notes,  and 
memoranda  obtained  from  SDC  personnel  who  participated  in  a  variety  of  capa¬ 
cities  in  the  BUIC  III  effort;  similar  types  of  documents  obtained  from  SDC?s 
BUIC  Management  File;  the  BUIC  III/SDC  Configuration  Index,  SDC's  BUIC  Monthly 
Management  Report  going  back  to  July,  1964,  SDC's  Technical  Objective  and 
Plan  series,  SDCfs  Air  Defense  Division’s  annual  Activities  Report,  SDC’s 
Lexington  Liaison  Office  Activity  Reports  and  a  variety  of  technical  documents 
(SDs,  TMs,  FNs,  Notes).  Important  sources  of  information  were  letters,  notes, 
and  memoranda  prepared  by  SDC  personnel  who  gathered  these  documents  while 
investigating  a  particular  problem  at  the  request  of  an  SDC  manager. 
Additionally,  extensive  use  was  made  of  the  files  of  SDC’s  System  Management 
Project,  which  has  provided  continuing  support  to  ESD  during  the  past  5-6 
years  in  developing  the  computer  program  acquisition  guidelines,  and  whose 
members  have  also  conducted  special  studies  associated  with  the  BUIC  II  and 
BUIC  III  programs. 

In  addition  to  the  document  sources,  much  information  in  this  report  has  been 
supplied  by  SDCers  who  participated  in  the  BUIC  III  program  and  provided  first¬ 
hand  data  on  the  application  of  the  375-series  to  the  design  and  development 
of  the  operational  computer  program  system  segment.  Material  for  Chapter  VI, 
Discussion  and  Evaluation,  was  initially  gathered  in  a  series  of  interviews 
and  group  meetings  with  SDC  technical  and  managerial  personnel  in  the  Military 
System  Division.  Their  number  was  too  large  to  permit  listing  of  all  of  their 
names.  These  people  not  only  provided  essential  basic  information  but  also 
contributed  useful  comments  and  corrections  as  the  writing  progressed  through 
several  drafts.  Helpful  factual  comments  were  also  supplied  by  several  members 
of  MITRE  Corporation. 

In  working  with  this  material  and  benefitting  from  the  many  comments  and  sug¬ 
gestions  received,  the  three  authors  wish  to  make  it  plain  that  the  interpreta¬ 
tions  and  viewpoints  expressed  in  the  report  are  their  own. 
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CHAPTER  II 


BACKGROUND  OF  SYSTEMS  MANAGEMENT  FOR  COMMAND  AND  CONTROL  SYSTEMS 


A.  THE  PROGRAM  DEFINITION  PHASE 

On  14  January  1963,  the  Office  of  the  Secretary  of  Defense  issued  a  draft 
instruction  entitled  "Program  Definition."  The  concept  of  "program  definition" 
was  based  upon  the  experience  of  AFSC’s  Ballistic  Systems  Division  (BSD).  The 
formal  presentation  of  the  concept  appeared  in  BSD  Exhibit  62-101,  System 
Analysis:  Procedures  for  System  Definition,  published  in  June,  1962,  for  the 

Mobile  Mid-range  Ballistic  Missile  (MMRBM)  program.  BSD  exhibits  had  pre¬ 
viously  been  forerunners  of  military  management  procedures  with  Air  Force-wide 
applicability  and  this  proved  to  be  the  case  in  this  instance.  The  BSD  exhibit 
proposed  a  new  "system  definition"  phase,  Phase  I,  to  cover  the  first  four 
months  of  the  Acquisition  Phase.  The  Department  of  Defense  instruction 
required  such  a  phase  for  all  large-scale  military  system  development  programs, 
not  merely  those  within  BSD  or  the  Air  Force. 

The  new  phase  was  defined  as  follows  in  the  draft  DOD  instruction:* 

"Program  Definition  is  the  formal  process  whereby  preliminary 
engineering  and  management  planning  are  accomplished  in  order 
to  arrive  at  definite  performance  specifications  and  refined 
cost  and  schedule  estimates  for  the  project  under  consideration. 

It  is  a  funded  effort  by  one  or  more  contractors,  working  in 
close  collaboration  with  the  government,  having  as  essential 
objectives  the  achievement  of  best  technical  approaches,  the 
identification  of  high  risk  areas,  and  the  development  of  an 
adequate  basis  for  firm  fixed  price  or  incentive  contracting 
of  the  main  body  of  a  major  development  project." 

Headquarters  USAF  responded  quickly  to  the  DOD  draft  instruction  and  then 
turned  the  matter  over  to  the  Air  Force  Systems  Command  for  detailed  study  „ 
Headquarters  AFSC  arranged  for  a  Command-wide  meeting  to  be  held  at  Andrews 
AFB  on  6  and  7  February  1963  to  review  the  impact  of  the  new  concept  on 
existing  management  policies  and  procedures.  Representatives  of  the  Depart¬ 
ment  of  Defense  as  well  as  the  Army  and  the  Navy  also  attended  and  participated. 


The  expression  "Program  Definition"  was  changed  to  "Project  Definition"  and 
subsequently  to  "Contract  Definition"  in  successive  revisions  to  the  original 
draft  instruction,  which  were  issued  in  February  1964  and  July  1965  as 
DOD  Directive  3200.9. 
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It  is  evident  in  the  report  of  the  Command-wide  meeting  that  the  cost  concerns 
of  the  Department  of  Defense  were  a  primary  factor  in  the  publication  of  the 
draft  instruction  on  Program  Definition.  It  was  closely  related  to  the 
experimentation  then  going  on  throughout  the  aerospace  industry  with  a  new 
management  tool  known  as  Program  Evaluation  Reporting  Technique  (PERT) .  A 
variation  of  PERT  called  PERT/Cost  was  clearly  exciting  to  DOD  personnel  since 
it  provided  "a  single  system  for  collating  the  three  things  that  one  has  to 
manage  in  a  program. .. the  work  to  be  accomplished,  the  time  schedule .. .and  the 
cost...."*  Mr.  Robert  S.  Tucker,  Assistant  Director  for  Engineering  Management 
in  Department  of  Defense  Research  and  Engineering  (DDR&E) ,  who  presented  this 
interpretation  of  the  DOD  position,  went  on  to  say  that  PERT/Cost  seemed  to  be 
a  powerful  tool  for  managing  large  development  programs  of  the  type  represented 
by  TITAN  III,  then  under  development.  At  this  point  it  is  worth  quoting  a 
later  section  of  the  report  on  PERT/Cost  in  full:** 

“About  two  years  ago  (1961)  when  people  really  began  to  worry  about 
bad  cost  estimates  and  the  spiralling  cost  of  weapon  systems,  the 
discussions  always  centered  about  the  very  beginning  of  the  program. 

It  was  stated  that  most  of  our  problems  were  caused  because  we 
hadn’t  had  a  good  cost  estimate  to  start  with.  Everybody  seemed  to 
agree  that  this  was  so.  They  further  agreed  that  you  couldn’t  get 
a  good  cost  estimate  unless  you  had  a  good  statement  of  work  to 
define  the  job  clearly.  We  began  by  thinking  along  the  lines  of 
how  we  could  get  a  better  statement  of  work  at  the  beginning  of 
a  program  and,  therefore  get  a  better  cost  estimate." 

The  objective  of  the  Program  Definition  Phase,  then,  was  to  obtain  accurate 
cost  estimates,  and  the  procedure  for  achieving  this  was  by  associating  costs, 
not  with  some  "combination  of  functions  and  hardware"  but  by  analyzing  the 
system  into  subsystems  and  then  into  specific  pieces.  These  pieces  are 
identified  as  "hardware  end-items"  and,  on  the  basis  of  these  end-items,  PERT 
networks  are  created.  This  approach  was  applied  during  the  TITAN  III  program 
in  the  belief  that  it  would  provide  "a  firm  basis  for  negotiation  of  definitive 
contracts  with  the  contractors  when  they  were  selected."***  In  this  way,  the 
contractor’s  cost  estimates  would  be  related  to  explicitly  defined  items  of 
work  instead  of  an  amorphous  work  statement.  Phase  I  of  the  Acquisition  Phase 
required  competition  among  contractors  for  the  award  of  development  contracts. 

It  was  believed  that  PERT/Cost  would  provide  a  basis  for  evaluating  competing 
proposals  from  industry. 


*  The  Program  Definition  Phase,  Report  of  a  Command-wide  Conference  held  at 
Hq.  AFSC,  Andrews  AFB,  6  and  7  February  1963,  p.  6. 

**  Ibid. ,  pp.  51-52. 

***  Ibid. ,  p.  7. 
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The  use  of  PERT/Cost  approach  in  the  TITAN  III  program  was  regarded  as  very 
successful  by  the  Department  of  Defense  and  it  asked  for  a  similar  approach  to 
the  development  of  BSDTs  Mobile  Mid-range  Ballistic  Missile  (MMRBM)  program 
then  just  getting  underway.  In  this  case,  DOD  referred  to  the  period  of 
system  definition  as  the  ’’Program  Definition  Phase”  and  that,  according  to 
Mr.  Tucker,  is  how  the  name  came  into  being. 

In  essence,  the  value  of  the  Program  Definition  Phase,  according  to  the  Depart¬ 
ment  of  Defense,  was  that  it  would  make  possible  the  identification  of  the 
components  of  a  large-scale  program  before  it  was  undertaken,  rather  than 
after  it  was  contracted  for,  and  it  would  provide  a  means  for  controlling  the 
program,  especially  costs,  after  it  was  begun. 

B.  IMPACT  OF  PROGRAM  DEFINITION  CONCEPT  ON  ESD 

Prior  to  the  publication  of  the  DOD  draft  instruction  on  Program  Definition 
and  the  AFSC-sponsored  Command-wide  meeting  in  February,  1963,  the  Electronic 
Systems  Division  had  initiated  efforts  to  improve  the  procurement  procedures 
for  large-scale,  computer-based,  command  and  control  systems.  In  August,  1962, 
personnel  representing  ESD,  MITRE  Corporation,  and  System  Development  Corpora¬ 
tion  met  at  Headquarters,  ESD,  and  established  a  Task  Group  to  examine  the 
problems  associated  with  the  acquisition  of  the  L-systems  which  were  ESDTs 
responsibility.  MITRE  and  SDC  were  not  for  profit  organizations  providing 
ESD  with  technical  support — MITRE  for  over-all  system  engineering  and  SDC 
for  computer  programs  and  associated  products,  a  ’’package”  referred  to 
at  that  time  by  the  generic  term  ’’software.” 

The  reasons  for  the  creation  of  the  Task  Group  may  be  briefly  summarized:  (1) 
pressure  from  the  DOD  and  USAF  for  tighter  and  more  effective  control  over  the 
procurement  process;  (2)  ESDTs  concern  over  the  adequacy  of  its  own  management 
process;  and  (3)  concern  by  the  ESD  SPOs  about  managing  a  procurement  process 
for  ’’software”  aspects  of  military  systems,  aspects  which  were  relatively 
unfamiliar  to  them  and  which  had  been  managed  up  to  that  time  by  the  con¬ 
tractors  . 

The  ESD/MITRE/SDC  Task  Group  was  composed  of  a  Steering  Committee  and,  over  a 
period  of  time,  a  number  of  ad^  hoc  Working  Groups.  The  Steering  Committee  was 
composed  of  senior  representatives  from  the  three  participating  organizations. 
Responsibilities  of  the  Committee  were  to  identify  problem  areas  that  needed 
investigation,  to  define  specific  tasks  for  the  Working  Groups,  to  provide  the 
groups  with  guidance,  and  to  review  their  work. 

The  first  Working  Group,  composed  like  the  Steering  Committee  of  representatives 
from  ESD,  MITRE,  and  SDC,  and  under  the  chairmanship  of  then  Lt.  Col.  S.  G. 
Morgan  of  ESD,  studied  the  acquisition  histories  of  465L,  473L,  425L,  416L, 
and  412L  to  attempt  to  identify  the  typical  problems  encountered  in  their 
management  and  to  assess  the  adequacy  of  the  management  techniques  used  to  deal 
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with  them.  The  Working  Group  reported  the  results  of  its  investigation  in 
October  1962.  The  following  areas  were  described  as  major  sources  of  problems 
in  need  of  additional  study  and  improvement: 

.  Management  tools  and  techniques,  i.e.,  specifications,  contract 

work  statements,  concurrence  documents,  etc.,  for  acquiring  computer 
programs,  associated  products,  and  related  technical  data. 

.  User  relationships  to  development  agencies  and  user  participation 
throughout  the  system  development  process. 

.  Capabilities  of  ESD  military  personnel  in  the  fields  of  system 
design  and  computer  programming. 

.  The  philosophy  and  concepts  of  system  design. 

A  second  Working  Group  was  created  to  address  itself  to  the  problem  areas 
identified  by  its  predecessor  with  a  particular  focus  on  the  first  area, 
management  tools  and  techniques.  The  task  assigned  to  the  Group  was  to  "pre¬ 
pare  drafts  or  outlines  of  ESD  Exhibits  which  would  serve  as  guidance  and 
illustration  in  the  use  or  applicability  of  specific  documentation,  procedural 
steps,  processes  and  definitions  for  the  uniform  conduct  of  management  of  com¬ 
puter  program  design  and  acquisition  . The  results  of  this  Working  Group 1 s 
efforts  were  published  on  18  February  1963  as  a  MITRE  document,  TM-3551, 

Computer  Program  Acquisition  Study.  This  document  constituted  a  good  beginning, 
despite  some  deficiencies,  in  what  was  to  become  a  long-range  task  of  detailed 
definition  of  the  procurement  process  for  computerized  systems.  It  identified 
the  major  problems  associated  with  the  acquisition  of  computer  programs  and 
it  recommended  possible  solutions  to  them. 

Meanwhile,  significant  events  were  also  taking  place  elsewhere  in  Systems 
Command.  Following  the  Command-wide  meeting  at  Headquarters  AFSC  in  February, 
1963,  General  Schriever  established  a  system  Definition  Task  Force  (SDTF)  on 
4  March  1963.  The  "charter"  for  the  SDTF  made  it  clear  that  the  AFSC  Commander 
considered  the  work  of  the  Task  Force  to  be  of  the  utmost  importance  and  to 
merit  top  priority  support  from  all  AFSC  divisions.  The  first  meeting  of  the 
SDTF  was  held  a  week  later  at  Headquarters  BSD,  which  had  been  made  the  lead 
division  for  the  task  at  hand.  An  interdivi sional  team  was  formed  as  a  result 
of  this  meeting  and  it  began  full  time  work  at  Norton  AFB,  California,  on 
1  April  1963.  Lt .  Col.  S.  G.  Morgan  was  the  ESD  representative. 

The  immediate  objective  of  the  Task  Force  was  to  incorporate  a  Definition  Phase 
into  the  typical  system  life  cycle,  in  compliance  with  General  Schriever* s 
directive  to  implement  the  DOD  instruction  on  that  concept,  and  to  revise 
existing  Air  Force  systems  management  documentation  accordingly.  However, 
after  studying  the  problem,  the  SDTF  undertook  the  reorganization  and 
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consolidation  of  the  entire  systems  management  framework.  This  task  took  the 
form  of  initiating  the  development  of  a  guidance  manual  for  general  use  by  the 
System  Program  Offices  throughout  AFSC,  which  would  contain  uniform  procedures 
covering  all  phases  of  the  system  life  cycle  and  all  major  aspects  of  systems 
management.  This  manual  (which  eventually  became  AFSCM  375-4)  would  presumably 
be  applicable  to  electronic  command  and  control  systems  as  well  as  to  jet  air¬ 
craft,  missiles,  and  space  vehicles. 

While  these  events  were  transpiring  at  the  Command  level,  the  activities  of 
the  ESD/MITRE/SDC  Task  Group  were  interrupted  by  the  receipt  of  a  letter  from 
AFSC,  dated  20  March  1963,  requesting  the  Division’s  comments  on  the  DOD  draft 
instruction  on  Program  Definition  and  the  BSD  Exhibit  62-101  on  the  same 
subject.  Senior  ESD,  MITRE,  and  SDC  representatives  met  to  consider  how  to 
reply. 

Various  staff  groups  were  asked  to  review  the  DOD  instruction  and  BSD  exhibit. 
The  consensus  was  that,  although  there  was  no  question  about  the  desirability 
of  a  "program  definition"  phase  to  precede  acquisition,  the  proposed  procedures 
were  based  upon  experience  with  weapons  systems  and  were  not  suitable  for 
command  and  control  systems.  Both  MITRE  and  SDC  recommended  to  ESD  that  it 
resist  any  attempt  to  establish  a  single  type  of  program  definition  procedure 
for  all  Air  Force  system  programs. 

One  of  the  products  of  the  Definition  Phase  identified  by  the  DOD  was  a 
detailed  PERT/Cost  network  "for  the  development  of  all  items  contained  in  the 
system  or  portion  thereof  on  which  he  (the  contractor)  was  asked  to  bid."  A 
list  of  all  end  items  was  to  be  included  and,  for  each  end  item,  performance 
specifications,  cost  estimates,  and  foreseeable  technical  problems  were  to  be 
presented.  In  retrospect,  the  dismayed  reactions  of  the  MITRE  and  SDC  staffs 
to  these  demands  upon  their  technical  virtuosity  are  not  difficult  to  under¬ 
stand.  Detailed  end  items  had  never  before  been  systematically  identified  in 
advance  of  development  for  the  then  current  crop  of  L-systems.  Disagreement 
was  rampant  over  the  kinds  of  products  which  should  be  included  under  the 
rubric  "software."  Did  it,  for  example,  include  "system  training,"  "human 
engineering,"  "system  evaluation,"  etc.?  No  reliable  cost  estimates  for 
computer  programming  tasks  in  military  command  and  control  systems  existed. 

SDC  personnel  could  not  see  how  computer  program  end  items  could  be  identified 
sufficiently  at  this  early  phase  in  the  system  life  cycle  to  provide  a  basis 
for  cost  estimates  accurate  enough  for  fixed  price  or  incentive  contracts — the 
basic  objective  of  the  DOD  instruction. 

Considering  the  computer  programming  state-of-the-art,  the  requirement 
to  identify  "foreseeable  technical  problems"  during  Phase  IB,  was  regarded 
by  experienced  programmers  as  an  unrealistic  one.  In  the  early  1960s, 
computer  programmers  regarded  a  large  part  of  what  they  did  in  military  systems 
as  research  or  exploratory  development.  The  DOD  requirements  implied  a  more 
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highly  developed  technology.  In  Phase  IB,  for  example,  it  was  stated  that  the 
contractors*  approach  ’’should  be  such  that  the  system  development  to  follow  can 
be  accomplished  essentially  as  an  engineering  task  requiring  little  or  no 
further  exploratory  investigation.”  In  general,  the  feeling  at  ESD  was  that 
the  DOD  concept  of  the  system  life  cycle  had  the  phases  in  a  reverse  order 
of  emphasis  insofar  as  L-systems  were  concerned.  For  L-system  development, 
more  time  should  be  devoted  to  system  design  and  relatively  less  to  acquisition 
They  saw  the  basic  problem  as  ’’the  proper  phasing  of  software  with  hardware 
design  and  development.”  Past  experience  indicated  that  requirements  for  the 
computer  programs  were  usually  underemphasized  during  the  early  stages  of  a 
system  program.  Prior  to  1963,  for  example,  it  was  common  for  computer 
equipment,  displays,  and  operator  consoles  to  be  contracted  for  before  the 
writing  of  performance-level  specifications  for  computer  programs  was  initiated 

In  spite  of  such  questions,  however,  the  obvious  importance  of  the  general 
problem  motivated  ESD  and  its  supporting  not-f or-prof its  to  take  immediate 
action. 

The  ESD/MITRE/SDC  Steering  Committee  established  the  third  Working  Group  at 
this  point  to  provide  technical  support  to  ESD  in  preparing  its  response  to 
AFSC,  and  to  provide  limited  support  to  the  interdivisional  team  which  was 
already  in  session  during  the  first  week  in  April.  Known  as  the  System 
Acquisition  Working  Group  (SAWG)  this  group  led  an  active  and  productive 
existence  for  a  period  of  about  six  months.  Its  primary  objective  was  to 
define  an  acquisition  .process  which  would  take  into  account  the  unique  needs 
of  electronic  system  development  as  viewed  in  the  ESD  management  framework. 

The  personnel  associated  with  SAWG  continued  to  provide  occasional  support 
to  the  interdivisional  task  force  during. that  period.  However,  the  factors 
of  remote  geography,  disparities  of  initial  orientations,  and  inherent 
complexity  of  the  problems  prevented  the  SAWG  efforts  from  being  materially 
influenced  by  the  command-wide  concepts  being  evolved  at  Norton,  which  sub¬ 
sequently  resulted  in  the  375-series  manuals. 

The  SAWG  resulted  in  two  published  reports  which,  although  issued  separately 
by  MITRE  and  SDC,  represented  complementary  treatments  of  the  total  system 
problem  as  viewed  at  that  time  in  the  ESD  context.  These  were:  SDC 
TM-(L)-LX-74/000/00  (Draft),  Command  Control  Software  Subsystem  Development 
During  the  Conceptual,  Program  Definition,  and  Acquisition  Phases,  14  August 
1963;  and  MITRE  TM-69,  The  Electronic  Systems  Acquisition  Process,  31  October 
1963. 

Both  reports,  in  different  ways,  were  something  less  than  finished  products. 
TM-69  defined  a  general  process  at  the  total  system  level,  in  terms  of  the 
four  subsystems  which  had  been  chosen  as  typical  of  an  electronic  system  — 
namely,  hardware,  software,  testware,  and  facilities.  While  the  intent  was 
to  amplify  processes  subsequently  for  the  subsystems  (except  software,  which 
was  amplified  in  the  SDC  report),  this  intent  was  never  realized.  The  SDC 
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document  was  published  only  in  draft  form.  Like  TM-69,  it  contained  a  variety 
of  management  concepts  and  assumptions  which  later  proved  to  be  incompatible 
with  the  Command-wide  375-series  philosophy.  However,  it  did  provide,  for 
the  first  time,  a  comprehensive  description  of  the  technical  processes 
involved  in  "software"  development  which  both  reflected  a  breadth  of  L-system 
experience  and  established  valuable  precedents  for  subsequent  work. 

C.  DEVELOPMENT  OF  CURRENT  CONCEPTS  AND  PROCEDURES 

The  SAWG  activities  terminated  in  late  1963.  However,  the, Command-wide 
efforts  which  were  destined  to  result  in  the  375-series  manuals  continued, 
with  increasingly  active  participation  by  ESD.  During  the  early  part  of  1964, 
the  attention  of  aerospace  industry  and  defense  agencies  was  drawn  to  the 
newly-revised  issue  of  AFSCM  375-1,  which  had  begun  to  expand  the  scope  and 
importance  of  configuration  management.  Although  the  immediate  impact  of  the 
manual  was  on  systems  and  items  of  equipment,  a  decision  had  already  been 
reached  by  Systems  Command  that  the  principles  of  configuration  management 
should  be  extended  to  cover  computer  programs.  In  effect,  this  was  a  decision 
that  computer  programs  are  a  class  of  system  components  which  should  be  managed 
like  items  of  equipment  —  as  opposed,  specifically,  to  items  of  data. 

The  decision  to  classify  computer  programs  with  equipment  for  purposes  of 
management  was  not  altogether  novel.  However,  appearing  as  it  did  in  the  con¬ 
text  of  the  emerging  375-series  concepts,  it  provided  a  new  set  of  criteria 
and  guidelines  for  approaching  questions  of  computer  program  acquisition  which 
had  not  been  present  in  the  earlier  ESD  studies.  Further  studies  were  clearly 
needed,  since  it  was  recognized  that  the  equipment  procedures  were  of 
questionable  applicability  to  computer  programs  in  many  areas,  not  only  within 
configuration  management  but  throughout  the  systems  management  structure.  As 
a  result,  and  because  of  ESD's  primary  interest  in  computer-based  systems,  ESD 
was  designated  the  lead  division  for  Systems  Command  to  resolve  the  associated 
problems . 

„  Since  that  time,  the  tasks  undertaken  have  been  concerned  with  developing  the 
necessary  adaptations  of  requirements  and  procedures  in  the  areas  of  configu¬ 
ration  management,  data  management,  testing,  and  system  engineering,  using 
the  established  manuals  covering  those  areas  as  points  of  departure.  This 
process  actually  began  in  late  1963,  as  a  result  of  a  tentative  proposal  made 
by  the  416M  SPO  to  apply  the  ANA  Bulletin  No.  445  to  SDC's  contract  for  the 
BUIC  II  computer  programs.  Problems  posed  by  the  bulletin's  equipment-oriented 
language  and  procedures  stimulated  SDC  to  undertake,  by  way  of  a  counter 
proposal,  to  develop  a  special  exhibit  which  would  apply  to  managing  computer 
program  changes.  A  series  of  drafts  of  the  exhibit,  initiated  in  December, 
1963,  evolved  into  a  final  exhibit  which  was  attached  to  the  BUIC  II  contract 
in  June  1964. 
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In  the  interim,  SDC  had  also  been  invited  by  the  Technical  Requirements  and 
Standards  Office  of  ESD  to  comment  on  the  revised  AFSCM  375-1  which  was  issued 
in  draft  form  in  January  1964.  Through  interest  in  the  general  problem,  SDC 
initiated  a  staff  study,  expanding  the  BUIC  II  exhibit  effort,  which  both 
influenced  the  content  of  the  BUIC  II  exhibit  along  lines  suggested  by  375-1 
and  also  resulted  in  a  report  (SDC  TM-1918,  1  June  1964)  containing  a  proposed 
approach  to  computer  program  configuration  management  for  general  application. 


In  January  1965,  the  Technical  Requirements  and  Standards  Office  took  steps 
to  initiate  continued  studies  on  a  more  formal  basis.  A  Software  Management 
Committee,  composed  of  representatives  from  a  number  of  ESD  offices,  was 
formed  to  coordinate  and  direct  the  efforts,  and  arrangements  were  made  for 
contract  support  by  SDC.  At  SDC,  the  contract  work  was  carried  out  by  a  small 
group  of  the  personnel  who  had  been  associated  with  the  earlier  work  in  SAWG 
and  configuration  management.*  The  contract  for  the  first  six-months  period 
was  sponsored  by  the  416M  SPO,  with  specific  objectives  to  (1)  develop  a  set 
of  AFLC/AFSC  Forms  9  covering  the  information  processing  data  items  for  a 
typical  system  program,  to  fill  a  vacuum  which  existed  in  that  area  in  the 
310-1  manual,  and  (2)  to  expand  and  refine  the  requirements  for  computer  pro¬ 
gram  configuration  management.  Requirements  incorporated  in  the  BUIC  III 
contract,  which  are  the  subject  of  this  report,  were  largely  products  of  that 
effort . 

Continued  work  by  the  ESD/SDC  team  during  the  next  few  years  was  devoted  to 
refining  those  early  products,  expanding  the  coverage  of  375-series  areas, 
and  coordinating  products  with  other  agencies.  Figure  2-1  summarizes  the 
events  which  have  been  described,  and  also  indicates  a  number  of  relevant 
events  which  have  occurred  since.  The  principal  documents  shown  on  the  chart 
beyond  those  already  discussed  are  identified  and  explained  briefly  below. 

ESD  416M-34  —  The  BUIC  III  configuration  management  exhibit  was  issued 
before  the  general  exhibit,  EST-1 ,  became  available.  It  was  based 
on  materials  resulting  from  the  project  described  above  (SDC  TM-1918/ 
01) ,  but  with  a  few  revisions  which  (1)  had  been  made  in  the  course 
of  subsequent  Air  Force/industry  coordination  and  (2)  were  indicated 
for  the  specific  BUIC  III  applications. 

EST-1  —  The  ESD  Exhibit  EST-1  was  first  issued  in  May  1966.  Based  on 
TM-1918/01,  it  was  reorganized  into  the  form  of  change  pages  and 
additional  exhibits  to  AFSCM  375-1. 


*  This  group  became  the  SDC  System  Management  Project,  which  has  been  in 
existence  since  that  time,  usually  at  a  level  of  2-4  members.  Current 
members  are  the  authors  of  this  report. 


10 


11 


Evolution  of  Systems  Management  Documentation 


EST-2  —  ESD  Exhibit  EST-2  was  directed  towards  the  "parent"  manual  of 
the  375-series,  AFSCM  375-4.  The  exhibit  contained  some  minor 
additions  to  the  375-4  flow  charts  of  events  for  a  system  life  cycle 
and  a  set  of  supplementary  narratives,  keyed  to  the  narratives  of 
375-4,  to  clarify  application  of  the  manual  to  ESD  systems.  The 
exhibit  was  revised  and  reissued  the  following  year  (1967)  in  the 
form  of  ESD  Supplement  1  to  AFSCM  375-4. 

EST-3  —  ESD  Exhibit  EST-3,  "Instructions  for  Conducting  Formal  Technical 
Reviews,  Inspections,  and  Demonstrations,"  was  written,  by  personnel  of 
the  Technical  Requirements  and  Standards  Office  to  clarify  the  subject 
topics  for  the  benefit  of  the  program  offices.  It  covers  those 
requirements  as  they  pertain  to  systems  and  equipment,  as  well  as 
to  computer  programs. 

TM-3596  —  This  document,  entitled  "System  Engineering  Guide  for  Computer 
Programs,"  was  initially  issued  in  September  1967  as  an  SDC  Technical 
Memorandum.  It  was  subsequently  reissued  as  ESD-TR-68-1,  in  March 
1968.  As  the  title  suggests,  it  was  directed  towards  the  areas 
covered  by  AFSCM  375-5.  However,  it  was  written  in  informational 
rather  than  directive  form,  to  provide  a  basis  for  further  study 
in  relation  to  the  applicability  of  375-5  requirements. 

POD  5010. 19/. 21  —  The  DOD  Directive  5010.19  and  Instruction  5010.21 
were  issued  in  July  and  August  of  1968,  providing  authority  for 
the  subsequent  issuance  of  military  standards  480  and  490  con¬ 
taining  uniform  configuration  management  requirements  for  use  by 
all  defense  agencies.  These  events  have  caused  the  Air  Force  to 
initiate  revisions  of  375-series  AFSC  manuals  to  comply  with  the 
new  standards,  and  will  result  in  some  variations  in  procedures 
and  terminology  from  those  described  during  later  chapters  herein. 

The  conversion  has  not  yet  been  accomplished  at  the  time  this  report 
is  published. 
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CHAPTER  III 


ORIGINS  AND  DESCRIPTION  OF  THE  BUIC  III  AIR  DEFENSE  SYSTEM 


A.  INTRODUCTION 

The  history  of  the  BUIC  system  program  is  treated  in  two  parts  in  this  and  the 
following  chapter.  This  chapter  reviews  the  evolution  of  BUIC  as  an  air 
defense  system  out  of  its  predecessor  systems:  the  Manual  Air  Defense  System 
and  the  Semi-Automatic  Ground  Environment  System  (SAGE) .  The  major  reasons  for 
the  creation  of  the  BUIC  III  system  are  reviewed  and  the  nature  of  the  system — 
its  operation  and  organization — are  described.  Chapter  IV  reviews  the  history 
of  SAGE  and  BUIC  II  in  terms  of  their  acquisition  as  system  programs  managed  by 
an  ESD  SPO. 

B.  MANUAL  AIR  DEFENSE  SYSTEM 

The  defense  of  the  continental  United  States  against  air  attack  was  not  regarded 
by  the  military  services  as  a  serious  problem  until  after  World  War  II.  Two 
events  altered  military  thinking  about  the  need  for  an  effective  continental 
air  defense:  the  successful  explosion  of  an  atomic  device  by  the  U.S.S.R.  in 
1949  and  the  outbreak  of  the  Korean  War  in  1950  which  raised  the  possibility 
of  an  attack  directly  against  the  United  States  mainland  by  manned  Soviet 
intercontinental  bombers. 

The  defense  of  the  continental  United  States  against  air  attack  in  the  early 
1950s  was  the  responsibility  of  the  Air  Defense  Command  (ADC)  of  USAF.  ADC 
operated  an  Aircraft  Control  and  Warning  (AC&W)  network  composed  of  a  few 
hundred  radar  stations  in  the  United  States  and  Alaska.* 

Responding  to  the  pressures  of  the  cold  war,  ADCfs  manual  AC&W  network  gradually 
expanded.  In  cooperation  with  Canada,  radar  coverage  was  extended  outside  of 
the  United  States  to  include  the  so-called  Pine  Tree  Line,  the  Mid-Canada  Line, 
and  the  Distant  Early  Warning  (DEW)  Line.  Radar  picket  ships  and  early  warning 
and  control  (radar)  aircraft  patrolled  the  approaches  to  the  east  and  west 
coasts  of  the  United  States,  thereby  extending  continuous  radar  coverage 
hundreds  of  miles  out  to  sea.  The  Greenland-Iceland-United  Kingdom  (GIUK)  Line, 
also  composed  of  airborne  radars,  extended  radar  coverage  across  the  north 
Atlantic.  The  Ballistic  Missile  Early  Warning  System  (BMEWS)  was  added  to 
provide  warning  against  a  possible  U.S.S.R.  ballistic  missile  threat. 


*  A  USAF  plan  in  1947  called  for  the  construction  of  411  radar  stations  at  a 
cost  of  about  $4,000,000. 
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The  primary  functions  of  air  defense  were  to  detect,  identify,  intercept,  and 
destroy  the  elements  of  an  enemy  air  attack.  The  operations  required  to  detect 
and  identify  aircraft  and  to  control  interceptors  were  carried  out  in  Air 
Defense  Direction  Centers,  the  basic  units  of  the  AC&W  network.  These  Direction 
Centers  (DCs)  were  located  at  the  surveillance  radar  stations  where  air  traffic 
was  kept  under  constant  observation. 

The  basic  operations  within  the  DC  may  be  briefly  described.  Operators  watched 
displays  of  radar  echoes  on  radar  scopes  for  signs  of  movement  indicating  the 
flight  of  aircraft.  After  detecting  an  aircraft,  the  operators  would  pass  its 
range,  azimuth,  and  estimated  speed  to  plotters  who  marked  its  position 
relative  to  local  geography  of  the  DC  on  a  large,  vertical,  Plexiglass  board. 
Other  plotters  maintained  the  current  status  of  interceptor  squadrons,  status 
of  winds  aloft,  and  status  of  the  air  defense  system  as  a  whole.  The  radar 
operators  kept  the  plotters  informed  of  the  movements  of  an  aircraft,  referred 
to  by  an  assigned  track  number,  as  it  flew  over  the  DCfs  geographical  area  of 
responsibility.  Each  track  was  quickly  identified  from  available  flight  plans 
or  was  declared  unknown  if  no  match  with  a  flight  plan  could  be  made.  Infor¬ 
mation  on  tracks  was  passed  laterally  to  adjacent  DCs  and  vertically  to  higher 
headquarters  known  as  Control  Centers  (CCs) ,  each  of  which  had  jurisdiction 
over  several  DCs. 

If  it  became  necessary  to  identify  an  unknown  track  visually  or  to  intercept 
a  potentially  hostile  aircraft,  the  Control  Center  issued  scramble  orders 
to  a  selected  interceptor  base  and  then  assigned  control  of  the  interceptor  to 
the  DC  best  located  to  direct  the  interception.  By  following  the  flight  of  the 
unknown  aircraft  and  the  interceptor  on  his  radar  scope  and  by  passing  verbal 
instructions  to  the  interceptor  pilot  via  radio,  a  weapons  director  guided  the 
fighter  pilot  to  a  point  where  visual  identification  of  the  unknown  aircraft 
could  be  made,  or,  if  the  unknown  was  determined  to  be  a  hostile  track,  into  a 
position  to  intercept  and  destroy  it. 

Organizationally,  each  DC  was  identified  as  an  AC&W  Squadron.  The  commanding 
CC  and  its  several  assigned  AC&W  squadrons  comprised  an  Air  Defense  Division. 

The  several  Divisions  into  which  the  United  States  was  divided  reported  to 
North  American  Air  Defense  Command  (NORAD)  headquarters  at  Colorado  Springs, 
Colorado.  In  1956,  the  continental  United  States  was  organized  into  twelve 
Divisions  grouped  into  three  Air  Defense  Forces. 

The  Manual  Air  Defense  System  suffered  from  two  basic  weaknesses:  inadequate 
radar  coverage  and  an  inability  to  process  with  sufficient  speed  and  accuracy 
the  large  amounts  of  data  which  could  be  expected  to  accompany  any  reasonably 
large-scale  air  attack.  The  Ground  Observer  Corps  was  utilized  to  attempt  to 
fill  in  the  gaps  in  radar  coverage.  In  response  to  the  problem  of  data 
processing  requirements  beyond  the  system’s  capabilities,  USAF  requested  the 
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assistance  of  the  Massachusetts  Institute  of  Technology.  In  1951,  a  special 
study  known  as  Project  Charles  was  instituted  by  MIT.  Recommendations  stemming 
from  Project  Charles  led  to  the  creation  of  the  Lincoln  Laboratory  which 
investigated  the  feasibility  of  using  an  electronic  digital  computer  for 
processing  radar  data.  It  was  this  investigation  and  other  similar  studies 
which  led  to  the  concept  of  a  Semi-Automatic  Ground  Environment  system  or  SAGE. 

Throughout  the  growth  and  development  of  SAGE,  selected  elements  of  the  manual 
system  have  continued  to  provide  a  back-up  capability  for  the  air  defense 
network. 

C.  THE  SAGE  SYSTEM 

SAGE  was  the  first  real-time,  computerized  air  defense  system  in  the  operational 
inventory  of  the  Air  Defense  Command.  Headquarters,  USAF  approved  its  acquisi¬ 
tion  in  1953  and  the  first  site  attained  operational  status  in  1958.  The 
rationale  for  the  replacement  of  the  Manual  Air  Defense  System  by  SAGE  is  reviewed 
in  these  words:* 

"In  early  1950,  the  military  concluded  that  the  manual  air- 
defense  system  in  use  at  that  time  could  not  adequately  co¬ 
ordinate  use  of  our  improved  hardware  against  the  growing 
enemy  threat.  The  capacity  of  the  system  was  too  low;  the 
speed  with  which  enemy  aircraft  could  be  detected,  tracked, 
and  intercepted  was  too  slow;  and  the  area  over  which  an  air 
battle  could  be  closely  coordinated  was  too  small.  The  problem 
was  one  of  inadequate,  nation-wide,  data-handling  capability: 
facilities  for  communication,  filtering,  storage,  control,  and 
display  were  inadequate.  A  system  was  required  which  would  1) 
maintain  a  complete,  up-to-date  picture  of  the  air  and  ground 
situations  over  wide  areas  of  the  country,  2)  control  modern 
weapons  rapidly  and  accurately,  and  3)  present  filtered  pictures 
of  the  air  and  weapons  situations  to  the  Air  Force  personnel 
who  conduct  the  battle." 

SAGE  performs  essentially  the  same  operational  air  defense  functions  as  did  its 
manual  predecessor:  surveillance,  identification,  and  weapons  control.  SAGE 
is  different  in  one  basic  respect — it  employs  electronic  digital  computers, 
in  addition  to  human  operators,  to  accomplish  selected  air  defense  functions. 

While  Air  Force  personnel  carry  out  such  tasks  as  the  initiation,  identification, 
and  monitoring  of  tracks,  the  computer  programs  plot,  correlate,  and  display 
surveillance  data.  The  assignment  and  commitment  of  weapons  are  made  by  weapons 


*  See  R.  R.  Everett,  et  al.,  p.  148,  in  the  bibliography 
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directors,  but  the  central  computer  continuously  tracks  the  position  of 
interceptors  and  makes  the  calculations  necessary  to  guide  a  weapon  to  its 
target.  Man-computer  communications  flow  is  shown  in  Figure  3-1.  A  key 
characteristic  of  SAGE,  deliberately  built  into  the  system,  is  its  semi¬ 
automatic  operations:  human  operators  have  the  capability  to  intervene  at 
practically  any  point  in  automatic  processes.  All  major  decisions  are  made  by 
Air  Force  personnel. 

The  SAGE  DC  contains  over  100  console  positions,  although  actual  manning  at  any 
given  time  varies  with  traffic  load  and  the  military  situation.  The  opera¬ 
tions  of  the  DC  are  conducted  in  several  rooms,  each  responsible  for  a  basic 
air  defense  function:  the  Surveillance  Room,  where  radar  returnfe  and  tracks 
are  monitored;  the  Weapons  Room,  where  interceptors  and  missiles  are  assigned  to 
targets  and  controlled;  the  Identification  Room,  which  processes  flight  plans; 
the  Manual  Inputs  Room,  where  data  received  by  voice  and  teletype  are  inserted 
into  the  computer;  the  Command  Post,  where  the  Battle  Staff  may  observe  the 
course  of  the  air  situation  or  battle  on  a  large  wall  display;  and  the  Training 
Battle  Staff  Room,  where  training  and  simulation  activities  are  conducted. 

The  SAGE  Direction  Center  (DC)  is  the  basic  operating  unit  of  the  system, 
modeled  after  its  manual  DC  prototype.  However,  communications  laterally  with 
adjacent  DCs  and  vertically  to  the  Combat  Centers  are  primarily  digitalized. 

The  elements  linked  to  the  SAGE  DC  by  both  automatic  and  manual  information 
processing  capabilities  are  shown  in  Figure  3-2.  These  elements  include  the 
command  element,  the  SAGE  CC;  adjacent  SAGE  and  manual  DCs;  the  Army  Air 
Defense  Command  Post  with  its  complement  of  Nike  guided  missiles;  interceptor 
bases;  weather  stations;  the  Air  Route  Traffic  Control  Center,  which  through 
its  Air  Movements  Information  Section  (AMIS)  provides  the  SAGE  DC  with  flight 
plan  information;  long  range  and  gap  filler  radars;  radio  transmitters  linking 
the  DC  by  voice  and  digital  data  to  interceptor  pilots;  BOMARC  guided  missiles; 
and,  at  various  times  in  the  past,  Navy  radar  picket  ships  and  Air  Force  early 
warning  and  control  aircraft. 

The  SAGE  system,  in  1966,  consisted  of  14  DCs  under  the  command  of  5  CCs,  one 
of  which  (Ottawa)  was  a  combination  CC/DC. 

Each  SAGE  DC  contains  duplexed  IBM  digital  computers,  designated  the  AN/FSQ-7. 
The  two  components  ensure  round-the-clock  operations.  The  following  descrip¬ 
tion  of  the  computer  system  and  computer  programs  as  of  1966  is  quoted  from 
H.  Sackman.* 

"The  computer  system  consists  of  the  central  computer,  the 

computer  programs,  magnetic  drum  buffer  storage,  six  magnetic 


*  See  H.  Sackman,  pp.  109-111,  in  the  bibliography.  Changes  to  SAGE  have,  of 
course,  occurred  since  this  description  was  written. 
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Figure  3-1.  Man-Computer  Communication  Flow  in  SAGE 
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Figure  3-2.  SAGE  Data  Flow 
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tape  units  on  each  side,*  and  a  real-time  clock.  The  central 
computer  is  a  general-purpose,**  binary,  parallel,  single¬ 
address  machine  with  a  32-bit  word  length.  The  magnetic  core 
memory  originally  included  8,000  registers,  later  expanded  to 
69,000.  Memory  cycle  time  is  6  microseconds,  and  average 
instruction  operating  time  is  close  to  10  microseconds,  per¬ 
mitting  up  to  100,000  dynamic  instructions  to  be  executed  per 
second. n 

"There  are  12  magnetic  drums,  each  with  12,288  32-bit  words, 
for  a  total  capacity  of  some  150,000  words.  The  magnetic  drums 
are  used  for  storage  of  the  operational  program,  system  status 
information,  and  for  in/out  buffer  data.  A  break  feature  permits 
central  computer  operations  to  continue  while  data  are  transferred 
from  core  memory  to  a  terminal  device.  For  example,  although 
325  microseconds  are  required  to  transfer  a  word  from  core  to 
tape,  only  six  microseconds  of  central  computer  time  are  used 
during  this  period  for  the  instruction  that  initiates  the 
transfer.  This  feature  is  most  important  for  a  real-time 
computing  system  since  considerable  time  is  consumed  in  input/ 
output  searching,  waiting  and  transfers.  Separate  read-write 
heads  permit  in/out  operations  between  external  sources  and 
buffer  storage  to  occur  independently  of  central  computer 
operation.  This  frees  the  central  computer  from  routine 
service  activities  and  permits  it  to  do  more  complex  jobs  more 
closely  tied  to  air  defense  operations." 

"The  operational  or  real-time  program  system  for  the  SAGE 
Direction  Center  contains  about  100,000  instructions.  It  is 
partitioned  into  some  40  sub-programs  to  handle  specific 
functional  areas.  Roughly  speaking,  the  operational  program 
is  approximately  equally  divided  in  size  between  four  general 
areas:  input /output  operations,  man-computer  communications, 

tracking  and  weapons  control,  and  miscellaneous  areas  including 
bookkeeping  (updating  tables)  and  simulation.  The  various  Q-7 
support  programs  involve  over  half  a  million  instructions. 

They  include  utility,  assembly,  recording,  data  reduction, 
simulation,  site  production,  and  JOVIAL  compiler  programs." 


*  Four  tape  units  were  standard  at  SAGE  sites. 

**  The  AN/FSQ-7  was  actually  designed  to  incorporate  special-purpose  features 
for  air  defense  application. 
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"There  are  additional  operational  and  support  programs  for  the 
AN/FSQ-8*  (Q-8  for  short)  used  at  higher  headquarters  at  SAGE 
Combat  Centers.  The  Q-8  computer  is  physically  identical  with 
the  Q-7  computer  as  it  was  originally  designed  with  an  8,000 
word  core  memory.  A  broader  definition  of  the  computer  program 
system  might  reasonably  include  contractor  activities  in  direct 
support  of  SAGE  training  and  operations,  activities  requiring 
the  use  of  SAGE  and  other  computers  and  associated  support 
programs.  Under  this  broader  definition,  the  grand  total  of 
SAGE-related  programs  incorporates  more  than  one  million  com¬ 
puter  instructions." 

The  SAGE  system  suffered  from  one  very  critical  flaw  —  it  was  extremely 
vulnerable  in  an  age  of  thermonuclear  missiles.  As  so  frequently  happens  in 
the  race  between  offensive  and  defensive  weapons  development,  USAF  had 
committed  itself  to  SAGE  before  the  advent  of  the  ICBM.  In  addition  to  the 
question  of  SAGE  survivability,  USAF  was  concerned  over  what  it  regarded  as 
unexpectedly  high  operational  and  maintenance  costs.  Numerous  studies  were 
conducted  in  the  1959-1961  period  in  an  attempt  to  find  solutions  to  SAGE 
inadequacies . 

In  the  1959-1960  period,  Super  Combat  Centers  were  proposed  and  evaluated  in 
an  effort  to  reduce  SAGE  vulnerability.  The  SCCs  were  conceived  as  hardened, 
underground  sites.  However,  the  SCC  concept  did  not  resolve  the  problem  of 
the  high  cost  of  air  defense.  Rather  than  build  expensive  underground  Combat 
Centers,  USAF  turned  in  1961  to  an  alternative  concept  —  a  decentralized 
dispersed,  less  vulnerable  and  less  expensive  system  which  would  provide  a 
back-up  capability  in  the  event  SAGE  facilities  were  destroyed. 

The  Back-Up  Interceptor  Control  or  BUIC  system  was  a  natural  outgrowth  of  SAGE 
modes  of  operation.  In  Mode  I,  normal  air  defense  operations  are  conducted 
by  the  SAGE  DCs.  If  the  active  computer  failed,  the  standby  computer  assumed 
operational  responsibility.  In  the  event  a  SAGE  DC  was  completely  disabled. 
Mode  II  operations  began.  In  this  mode,  adjacent  DCs  expanded  their  geo¬ 
graphical  areas  of  responsibility  to  cover  the  disabled  sector.  Mode  III 
referred  to  an  operational  situation  in  which  manual  or  back-up  sites  assumed 
the  task  of  air  defense  when  their  SAGE  DCs  were  inoperative. 


D.  THE  BUIC  CONCEPT 

In  1961,  a  Task  Group  at  Headquarters,  USAF  developed  the  concept  of  an 


*  The  Q-8s  have  since  been  phased  out. 
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austere  interceptor  control  system  as  a  back-up  to  SAGE  and  submitted  their 
plans  to  NORAD,  ESD,  and  ADC.  The  cost  of  the  system  was  not  to  exceed  $100 
million,  using  off-the-shelf  equipment,  available  communications,  and  was  to 
be  acquired  in  the  shortest  possible  time.*  The  USAF  planners  believed  that 
the  BUIC  concept  would  make  a  SAGE  reconfiguration  possible  in  which  the  most 
vulnerable  DCs  would  be  eliminated  and,  as  a  result,  USAF  would  realize 
significant  savings  in  operations  and  maintenance  costs.  In  January  1962,  ADC 
published  an  operational  plan  for  BUIC  and  a  month  later  ESD  published  a 
document  dealing  with  BUIC  functional  design  and  performance  requirements. 

These  early  documents  recommended  the  creation  of  back-up  control  centers, 
called  NORAD  Control  Centers  (NCCs) ,  which  would  take  over  the  air  defense 
responsibilities  in  the  event  SAGE  control  capabilities  were  lost.  The 
assumption  that  the  NCCs  would  have  a  higher  probability  of  surviving  a  missile 
attack  than  SAGE  DCs  was  based  on  the  fact  that  they  were  to  be  collected 
with  selected  long-range  radar  sites  which  were  not  near  expected  ICBM  targets. 

The  BUIC  system  implementation  was  divided  into  three  phases.  BUIC  I  was 
conceived  as  an  interim  system  consisting  of  27  manual  NCCs.  This  system 
provided  an  immediate  manual  back-up  control  capability  comparable  to  the 
earlier  Manual  Air  Defense  System.  While  some  of  these  BUIC  I  NCCs  will 
remain  part  of  the  air  defense  system,  most  will  be  either  phased  out  or  con¬ 
verted  into  semi-automatic  NCCs.  BUIC  II  was  originally  planned  to  have  34 
NCCs;  however  budget  limitations  and  other  factors  reduced  that  number  to  13 
computerized  NCCs  capable  of  assuming  the  SAGE  air  defense  functions.  The  BUIC 
NCCs  do  not  have  the  same  data  processing  capabilities  as  SAGE  DCs,  but  perform 
basically  the  same  air  defense  operations  in  a  similar,  computer-assisted  manner. 
BUIC  II  is  currently  operational.**  The  BUIC  II  system  with  improved  and  ex¬ 
panded  capabilities,  roughly  twice  that  of  BUIC  II,  is  gradually  replacing  it. 

E.  THE  BUIC  III  SYSTEM*** 

BUIC  III  is  a  command  control  system  composed  of  semi-automatic  NCCs  located 

at  selected  long-rage  radar  sites.  One,  two,  or  three  NCCs  are  installed 

in  an  Air  Defense  Division  to  serve  as  a  backup  to  the  SAGE  DC  (see  Figure  3-3) . 


*  See  W.  S.  Melahn,  p.  1,  in  the  bibliography 

**  A  description  of  the  BUIC  II  system  may  be  found  in  a  System  Development 
Corporation  document,  BUIC  II  Interface  Exercise  Manual,  TM-2329/000/02 , 

4  April  1966. 

***  The  information  on  BUIC  III  in  this  section,  with  the  exception  of  Figure 

3-3,  is  taken  in  its  entirety  from  a  System  Development  Corporation  document, 
Introduction  to  BUIC  III  and  SETE,  TM-3000/100/02 ,  1  February  1968. 


21 


22 


BUIC  III  Sites 


When  the  DC  in  a  Division  is  operating,  the  NCCs  in  that  Division  monitor  the 
air  defense  situation  to  be  able  to  assume  air  defense  responsibility  quickly 
in  case  the  DC  becomes  inoperative.  This  operational  mode,  called  the  Monitor 
mode,  is  the  normal  operating  state  of  NCCs.  If  a  SAGE  DC  becomes  inoperative, 
the  NCCs  in  that  Division  change  to  an  Active  mode  of  operation  and  assume 
responsibility  for  air  defense  operations  within  the  Division.  At  the  NCC, 

BUIC  III  operators  receive  displays  of  filtered  and  processed  data  and  enter 
instructions  at  display  consoles  connected  to  the  BUIC  computer. 

A  digital,  teletype,  and  voice  communications  network  interconnects  NCCs  and 
other  air  defense  system  elements.  Most  of  the  data  entering  the  NCC  are  in 
digital  form  and  are  entered  directly  into  the  computer.  ‘Other  data  received 
by  voice  or  teletype  may  be  entered  into  the  computer  manually  by  card  inputs. 
The  NCC,  in  either  a  Monitor  or  an  Active  mode,  receives  a  continuous  flow  of 
data  which  must  be  processed  and  displayed  to  support  its  air  defense  role. 
There  is  also  a  continuous  flow  of  data  out  of  the  NCC,  but  the  volume  of  out¬ 
puts  is  much  less  in  the  Monitor  mode  than  in  the  Active  mode  when  the  NCC  has 
active  control  of  air  defense  elements  in  its  area  of  responsibility.  The 
major  elements  that  a  typical  NCC  interfaces  with  are  depicted  in  Figure  3-4. 

In  the  Monitor  mode,  an  NCC  maintains  a  current  air  picture  by  means  of  radar 
data  received  directly  from  radar  sites  and  track  data  received  from  its 
parent  DC  (the  DC  in  its  Division)  or  from  associate  NCCs  (other  NCCs  in  the 
same  Division) .  Radar  data  come  from  the  same  network  of  radars  used  by  the 
SAGE  system.  This  network  includes  long-range  radars  (LRRs) ,  gap-filler 
radars  (GFRs)*  and  airborne  long-range  radar  inputs  (ALRIs) .  These  data  are 
processed  and  displayed  at  the  NCC  in  the  same  manner  in  either  the  Monitor 
or  the  Active  mode.  An  NCC  in  the  Monitor  mode,  however,  has  no  control  over 
the  operations  at  the  radar  sites  -  In  the  Monitor  mode,  the  track  data  from 
the  DC  are  automatically  updated  by  the  BUIC  computer  program  by  means  of 
radar  inputs.  This  helps  to  ensure  the  continuity  of  tracks  in  the  system  in 
the  event  the  NCC  has  to  take  over  the  air  defense  function.  While  in  the 
Monitor  mode,  an  NCC  does  not  initiate  tracks,  request  height,  or  identify 
live  tracks,  nor  does  it  commit  weapons. 

When  the  parent  DC  becomes  inoperative,  the  NCCs  in  the  Division  take  over  the 
functions  of  the  DC.  The  Active  mode  NCCs  expand  their  interfaces  and  each 
assumes  control  of  a  predesignated  portion  of  the  Division.  External  defense 
elements  are  informed  by  voice  that  the  NCC  is  assuming  control  and  circuits 
are  established  for  the  flow  of  information.  An  NCC  replaces  its  parent  DC 
in  the  lateraltell  network  of  DC  communications.  One  NCC  in  the  Division 
becomes  interconnected  with  the  SAGE  Combat  Center  (CC)  which  is  responsible 


*  All  but  one  of  the  GFRs  have  been  phase  out. 
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for  the  direction  and  coordination  of  air  defense  functions  in  the  Region. 

The  NCC  transmits  track  data  and  weapons  status  information  to  the  CC  via  data 
link.  By  voice,  the  CC  provides  the  Active  NCC  with  early  warning  information, 
nuclear  detonation  reports,  and  information  on  adjacent  Region  activity.  If 
the  CC  becomes  inoperative,  and  no  DC  in  the  Region  remains  in  operation,  the 
Battle  Commander  in  a  designated  NCC  establishes  voice  communications  with  the 
NORAD  Combat  Operations  Center  (COC)  in  order  to  transmit  Region  status  data, 
battle  damage,  and  track  information  to  the  COC  and  to  receive  threat  infor¬ 
mation,  nuclear  hazard  data,  intelligence  data,  and  battle  direction. 

An  Active  NCC  is  responsible  for  all  air  surveillance  and  weapons  control 
functions  in  its  area  of  responsibility.  Close  coordination  with  radar  site 
personnel  is  needed  to  maintain  the  best  possible  surveillance  radar  data. 
Height  finding  radars  at  these  sites  are  also  controlled.  Tracks  are  initiated 
and  monitored  by  air  surveillance  personnel  at  the  NCC.  Identification 
responsibilities  are  assumed.  Flight  plan  information  received  from  the  Air 
Movements  Information  Section  (AMIS)  via  teletype  is  inserted  into  the  computer 
by  card  input  for  manual  and  automatic  identification  processing.  Weapons 
personnel  commit  manned  Interceptors  or  BOMARC  missiles  against  tracks 
identified  as  Hostile  or  the  tracks  are  passed  to  Army  Air  Defense  Command  Post 
(AADCPs)  for  engagement  by  ADA  fire  units.  Manned  interceptors  are  scrambled 
by  means  of  voice  communications  between  NCC  weapons  personnel  and  personnel 
at  the  Combat  Alert  Center  (CAC)  at  an  airbase.  Control  of  the  aircraft  is 
ordinarily  by  means  of  data  link  transmitted  via  Time  Division  Data  Link 
(TDDL)  but  two-way  voice  communications  are  provided  by  ground-to-air  voice 
radio  sites.  BOMARC  control  is  always  via  TDDL. 

In  the  remainder  of  this  chapter,  MCC  equipment,  computer  programs,  and 
personnel  are  described  briefly. 

1 .  Equipment 

Data  processing  and  display  equipment  are  installed  at  each  BUIC  II  NCC  to 
support  the  NCC  air  defense  role.  As  an  integrated  group  this  equipment,  the 
AN/GSA-51A  Radar  Course  Directing  Group,  is  capable  of  performing  the  calcula¬ 
tions  and  data  manipulations  required  by  computer  programs;  accepting  and 
transmitting  information  via  digital  data  circuits;  displaying  information 
to  operators;  accepting  and  processing  manually  inserted  information,  operator 
requests  and  commands;  and  storing  information  for  subsequent  processing. 

a.  Data  Processing  Equipment 

The  BUIC  III  data  processor  consists  of  solid  state  computers,  core  memories, 
and  input/output  modules.  Their  functions  and  the  functions  of  related  peri¬ 
pheral  equipment  are  described  below. 

Digital  data  computers.  Two  digital  data  computer  modules  interpret  and 
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execute  program  instructions.  They  operate  independently  and  each  is  capable 
of  controlling  the  operation  of  a  stored  program. 

Core  memories.  Eight  core  memory  modules  provide  a  high-speed  random  access 
storage  of  a  total  of  32,768  words. 

Controller-comparators .  Four  controller-comparators  or  input/output  control 
modules  control  the  transfer  of  data  between  the  core  memories  and  peripheral 
devices . 

Magnetic  drums.  Three  magnetic  drums  are  used  for  the  storage  of  computer 
programs,  large  blocks  of  data,  and  information  to  be  displayed.  Each  drum 
provides  65,536  words  of  storage.  Three  magnetic  drum  controller-converter 
enable  the  transfer  of  data  from  a  display  drum  to  a  data  display  console. 

Message  processors.  Two  message  processors  receive  digital  data  from  external 
sources,  store  the  data,  and  send  them  to  a  controller-comparator.  They  also 
store  output  messages  from  the  controller-comparators  prior  to  transmission 
over  the  output  circuits. 

Magnetic  tape  units.  Four  magnetic  tape  units  or  recorder-reproducers  are 
used  to  record  data  and  store  computer  programs  and  simulated  data.  They  are 
made  compatible  with  the  controller-comparator  modules  by  means  of  a  recorder- 
reproducer  controller. 

Teleprinter  set.  An  on-line  teleprinter  produces  permanent  records  of  selected 
data.  This  high-speed  device  can  print  120  characters  per  line  in  quadruplicate 
at  a  rate  of  600  lines  per  minute. 

Keypunch .  A  keypunch  machine  is  used  to  punch  and  verify  data  on  80-column 
cards . 

Punch  card  reader .  An  on-line  punch  card  reader  is  used  for  the  manual  inser¬ 
tion  of  information  to  be  used  by  the  computer  program. 

Teletypewriters .  Teletypewriters  (teletypes)  are  used  to  receive  flight  plan 
data,  winds  aloft  data,  and  weather  information.  They  output  information  In 
both  punched  paper  tape  and  printed  form. 

047  Tape-to-card  converter.  This  device  converts  the  punched  tape  output  by 
the  teletype  to  punched  cards  which  are  ready  for  entry  into  the  card  reader. 

Typewriter-punch  reader.  An  on-line  typewriter-punch  reader  provides  a  means 
of  communication  between  the  data  processor  and  maintenance  personnel. 
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Status  display  console.  A  status  display  console  is  provided  in  the  data 
processing  area  for  use  by  maintenance  personnel  in  controlling  and  monitoring 
all  central  and  peripheral  data  processing  equipment. 

Simulator  group.  The  simulator  group  consists  of  two  manual  input  keyboards 
located  in  the  data  display  area  and  one  compatibility  unit  in  the  data  pro¬ 
cessing  area.  This  equipment  can  be  used  to  generate  and  control  simulated 
interceptors  and  to  simulate  pilot  response  during  a  training  exercise. 

b.  Data  Display  Equipment 

Data  display  consoles  provide  operators  with  the  computer-generated  displays 
they  need  to  perform  their  assigned  tasks,  a  means  of  selecting  the  information 
to  be  presented,  and  a  means  of  inserting  requests  and  commands  into  the  com¬ 
puter.  Eleven  data  display  consoles  are  provided  in  those  NCCs  having  an  Air 
Defense  Artillery  responsibility.  All  other  NCCs  have  ten.  All  consoles 
are  logically  and  electrically  identical,  but  they  may  be  assigned  to  different 
functions  by  means  of  manual  input  card  messages.  A  typical  configuration  of 
display  equipment  and  assignments  is  shown  in  Figure  3-5.  The  minimum  and 
maximum  number  of  consoles  that  may  be  assigned  to  various  functions  is  as 
follows : 

Console  Function  Min.  Max. 


Senior  Director  1  1 

Weapons  Director(s)  0  6 

Air  Defense  Artillery  Director(s)  0  2 

Radar  Inputs  and  Countermeasures 
Officer/Air  Surveillance  Officer  1  1 

Air  Surveillance  Operator (s)  1  7 

Identification  Operator  (s)  0  2 

Simulation  Supervisor  0  1 

Flight  Path  Simulator (s)  0  3 

Target  Monitor  0  1 


Situation  Display.  The  data  display  console  contains  a  cathode  ray  tube  for 
the  display  of  information  concerning  the  air  defense  situation.  It  has  a 
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Figure  3-5.  Typical  BUIC  III  Display  Area 
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viewing  area  12  5/8  inches  square.  Symbols  and  vectors  are  used  to  display 
such  information  as  radar  data,  track  information,  alarms,  intercept  points, 
the  location  of  air  defense  facilities,  and  geographic  boundaries.  By  push¬ 
button  action  an  operator  may  select  display  expansion  levels  of  two,  four,  or 
eight  in  addition  to  the  basic  display  which  covers  an  area  1500  nautical  miles 
square . 

The  classes  or  categories  of  data  displayed  on  a  situation  display  may  be 
selected  to  be  appropriate  to  the  function  being  performed  at  each  individual 
console.  Forty  three-position  category  selection  switches  are  provided  on  the 
left  side  of  the  situation  display  viewing  surface  for  this  purpose. 

Tabular  display.  A  separate  cathode  ray  tube  is  provided  on  the  data  display 
console  for  the  display  of  data  in  a  tabular  format.  It  has  a  viewing  area  3 
3/6  inches  square.  Within  this  area  a  maximum  of  17  columns  and  16  rows  of 
symbols  may  be  displayed. 

Computer  alarms.  One  visual  and  one  audible  computer  alarm  are  provided  at 
each  console.  Activation  and  resetting  of  the  alarms  are  under  the  control 
of  the  computer  program  but  the  audible  alarm  may  also  be  reset  manually  by  the 
console  operator. 

Manual  intervention .  By  means  of  pushbuttons  or  switches,  an  operator  may 
enter  and  transmit  information  from  the  display  console  to  the  data  processor. 
The  switch  panel  on  the  console  contains  a  twelve-column  pushbutton  keyboard, 
a  rotary  heading  switch  for  the  insertion  of  heading  information,  and  an 
activate  pushbutton/indicator  to  transmit  information  to  the  data  processor. 

Light  sensor.  Operators  are  provided  with  light  sensor  devices  similar  to  the 
SAGE  light  guns  to  initiate  requests  or  commands  to  the  data  processor.  This 
device  is  activated  by  positioning  it  over  a  displayed  symbol  or  vector  by 
means  of  a  projected  circle  of  light. 

Coordinate  data  monitor.  A  coordinate  data  monitor  console  is  provided  for 
the  monitoring  of  radar  inputs.  This  equipment  is  also  called  a  Random  Access 
Plan  Position  Indicator  (RAPPI) . 

2 .  Computer  Programs 

The  computer  programs  provided  for  BUIC  III  were  identified  and  developed  as 
four  separate  computer  program  contract  end  items  (CPCEIs) ,  distinguished  on 
the  basis  of  system  functions  as  follows:  (a)  operational  functions,  (b) 
system  exercising  functions,  (c)  utility  functions,  and  (d)  computer  mainten¬ 
ance-diagnostic  functions. 

a.  Air  Defense  Computer  Program  (ADP) 

The  ADP  performs  the  calculations  and  data  manipulations  required  for  air 
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defense  operations.  It  provides  the  capability  for  tracking  and  identifying 
aircraft,  for  the  scrambling  and  control  of  interceptors,  and  for  the  transfer 
of  information  to  adjacent  facilities  or  higher  echelons.  In  addition,  a 
capability  is  provided  for  periodically  recording  operations  data  for  later 
analysis.  For  training  operations  crews,  a  means  of  simulating  certain  inputs 
to  an  NCC  is  included  in  the  program. 

b.  System  Exercise  Computer  Program  (SEP) 

The  SEP  provides  the  capability  to  generate  simulated  data  on  magnetic  tapes 
to  be  used  as  inputs  to  ADP  during  air  defense  exercises  and  during  the 
testing  of  ADP.  It  also  provides  a  means  for  analysis  of  data  recorded  during 
exercise  and  training  missions.  Summary  information  may  be  provided  on 
individual  operator  performance  as  well  as  on  the  performance  of  groups  of 
operators  or  of  the  system  as  a  whole. 

c.  Utility  Computer  Program  (UCP) 

The  UCP  is  the  system  used  for  the  production  and  maintenance  of  ADP  and  SEP. 
It  is  also  capable  of  maintaining  itself.  In  producing  a  coded  system,  infor¬ 
mation  is  accepted  by  the  program,  assembled  into  a  machine  language  and 
placed  on  a  magnetic  tape  for  later  use.  These  tapes  may  be  changed  via  the 
UCP  to  correct  errors  or  to  incorporate  required  changes  to  the  system, 

d.  Maintenance  Computer  Program  (MCP)* 

The  maintenance  computer  program  is  made  up  of  two  parts:  the  backup  con¬ 
fidence-diagnostic  (BCDP) ,  and  the  active  confidence-diagnostic  (ACDP) .  BCDP 
consists  of  confidence  checking  and  diagnostic  testing  programs  which  exercise 
and  operate  in  the  backup  equipments.  This  program  operates  in  parallel  with 
ACDP  or  ADP.  ACDP  operates  in  the  active  equipment  modules  when  ADP  is  not 
running.  It  provides  executive  control  for  parallel  BCDP/ACDP  operation, 
diagnostic  testing,  and  error  analysis.  The  MCP  provides  the  BUIC  III  system 
with  early  malfunction  detection  and  fault  isolation  to  minimize  the  effect 
of  equipment  failures  on  overall  system  performance.  The  computer  operators 
are  kept  informed  of  equipment  failures  through  audible  alarms,  visible  dis¬ 
plays,  or  printed  output  on  the  typewriter-punch  reader  or  the  teleprinter. 

3.  Personnel 


Operations  and  maintenance  personnel  are  stationed  at  each  NCC  for  the  perfor¬ 
mance  of  air  defense  operations.  They  are  divided  into  functionally  oriented 


*  MCP  is  a  Burroughs  Corporation  computer  program. 
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groups  consisting  of  a  Supervisory  Team,  an  Air  Surveillance  Team,  a  Weapons 
Direction  Team,  a  Manual  Data  Team,  a  Maintenance  Team,  and  several  other 
support  teams.  The  authorized  positions  which  constitute  these  teams  are  out¬ 
lined  in  the  following  paragraphs,  but  the  number  of  operational  positions  which 
will  be  manned  at  any  given  time  will  depend  on  the  NCC  status  and  the  Division 
air  defense  conditions. 

a.  Supervisory  Team 

The  Supervisory  Team  is  composed  of  the  Battle  Commander  (BC) ,  a  Senior 
Director  (SD),  and  a  Senior  Director  Technician  (SDT) .  The  BC  is  responsible 
for  the  NCC  operation  and  coordinates  with  higher  echelons  of  command.  The 
BCfs  position  is  at  a  table  in  the  display  area  adjacent  to  the  SD.  He  does 
not  have  a  display  console,  but  he  may  share  the  SD's  console.  The  SD  is 
responsible  for  air  defense  functions  being  performed  within  the  NCC  and 
coordinates  defense  operations  with  other  NCCs.  The  BC  is  supported  by  the 
Weather  Officer  (WXO)  and  the  Communications-Electronic  Staff  Officer  (CEO) , 
who  are  located  at  the  same  table  as  the  BC. 

b.  Air  Surveillance  Team 

The  Air  Surveillance  Team  is  responsible  for  radar  input  data  control,  tracking, 
height  finding,  track  identification  and  coordination  with  other  control 
centers.  The  team's  supervisor  is  responsible  for  dual  functions.  As  the 
Radar  Inputs  and  Countermeasures  Officer/Air  Surveillance  Officer  (RICMO/ASO), 
he  supervises  the  action  of  the  air  surveillance  function,  monitors  radar 
input  data,  and  coordinates  ECCM  activities  in  an  ECM  environment.  He  also 
supervises  the  manual  inputs  function.  The  Air  Surveillance  Technician  (AST) 
assists  him  in  these  duties.  Other  team  positions  include  Air  Surveillance 
Operators  (ASOs)  who  initiate  and  drop  non-interceptor  tracks,  monitor  active 
tracing,  initiate  height  requests,  and  monitor  passive  tracking;  and  Identi¬ 
fication  Operators  (IDOs)  who  identify  tracks  and  coordinate  flight  plan 
information  with  input  agencies. 

c.  Weapons  Team 

The  Weapons  Team  is  responsible  for  directing  and  controlling  air  defense 
weapons  and  coordinating  activities  with  other  control  centers  and  AADCPs . 

The  team  consists  of  Weapons  Directors  (WDs)  and  their  technicians,  and  an 
Air  Defense  Artillery  Director  (ADAD)  and  his  technician.  Army  personnel 
are  stationed  at  the  NCC  to  perform  ADA  functions. 

d.  Manual  Data  Team 

The  Manual  Data  Team  is  responsible  for  handling  data  received  by  voice  or 
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teletype  from  external  sources  and  entering  it  into  the  NCC  data  processor. 
Normally,  the  team  is  composed  of  three  Manual  Inputs  Operators  (MIOs)  and  a 
Manual  Status  Clerk  (MSC) .  The  duties  of  the  MSC  include  the  maintenance  of 
current  status  data  on  a  vertical  plotting  board  in  the  display  area.  Other 
than  this  task,  the  functions  of  the  Manual  Data  Team  are  performed  in  the 
Manual  Data/Weather  area,  which  is  a  91  by  12 1  space  partitioned  within  the 
data  processing  area.  This  team  has  no  data  display  console. 

e.  Weather  Team 


The  Weather  Team  provides  weather  information  to  operations  personnel  in  the 
NCC.  Weather  staff  support  is  given  to  the  Battle  Commander.  Nuclear  fallout 
patterns  are  developed,  and  weather  forecasts  and  winds  aloft  data  are  prepared 
for  the  Manual  Data  Team  to  enter  into  the  data  processor.  The  Weather  Team 
consists  of  a  Weather  Officer  (WXO)  whose  normal  position  is  at  the  BCfs  table 
in  the  front  of  the  display  area  and  a  Weather  Observer  (WXOB)  who  is  located 
in  the  Manual  Data/Weather  room  in  the  data  processing  area. 

f.  Maintenance  Team 


The  Maintenance  Team  is  under  the  direction  of  the  Electronic  Computer  Main¬ 
tenance  Office  (ECMO)  who  normally  supports  the  SD  in  the  display  area.  The 
Computer  Maintenance  Supervisor  (CMS)  and  the  Facility  Maintenance  Monitor 
(FMM)  have  over-all  responsibility  for  the  maintenance  and  operation  of  the 
data  processing  and  display  equipment.  They  are  assisted  by  Computer  Main¬ 
tenance  Technicians  (CMTs)  and  Display/Input-Output  Technicians  (D/IOTs).  A 
Computer  Program  Maintenance  Team  (CPMT)  is  responsible  for  monitoring  the 
operation  of  the  NCC  computer  programs  and  making  on-site  modifications  and 
corrections  as  necessary. 

g .  Target  Monitoring  Team 

The  Target  Monitoring  Team  consists  of  an  Exercise  Director  and  a  Target 
Monitor.  They  are  responsible  for  controlling  exercise  target  aircraft  and 
for  ensuring  the  safety  of  all  aircraft  during  air  defense  exercises  using 
live  aircraft.  These  are  not  full-time  positions  but  will  be  filled  by 
qualified  operations  personnel  as  a  supplement  to  their  normal  duties. 

h .  Simulation  Team 

The  Simulation  Team  is  responsible  for  the  generation  of  the  simulation  environ¬ 
ment  and  performance  evaluation  during  a  training  mission.  The  only  full  time 
position  is  the  Training  Technician  (TT) .  Other  positions  are  manned  as 
required  by  available  NCC  personnel. 
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CHAPTER  IV 


INFORMATION  PROCESSING  ELEMENTS 
OF  THE  SAGE/BUIC  II  SYSTEM  PROGRAMS 


A.  INTRODUCTION 

Whereas  the  preceding  chapter  presented  an  outline  description  of  the  SAGE/ 

BUIC  systems,  the  purpose  of  this  chapter  is  to  review  those  aspects  of  the 
earlier  system  programs  which  are  relevant  to  an  appreciation  of  the  BUIC  III 
experience.  As  in  other  portions  of  the  report,  the  focus  of  interest  is  on 
the  elments  of  contractor  responsibility  which  were  once  known  as  "software" 
and  are  presently  referred  to  as  the  computer  program,  or  information  pro¬ 
cessing,  system  segment. 

The  distinction  is  being  made  herein  between  a  system,  on  the  one  hand,  and  a 
system  program  on  the  other.  By  system  program  we  refer  to  the  management 
structure  and  organizational  activities  which  are  involved  in  the  process  of 
creating,  delivering,  and  sustaining  an  operable  and  supportable  system  (cf. 

AFR  375-1).  The  system  itself  is  the  collection  of  equipment,  computer  pro¬ 
grams,  facilities,  procedural  data,  and  personnel  which  is  provided  to  a  user 
for  operational  employment.  Thus,  in  brief,  the  system  i,s  the  product,  while 
the  program  is  the  set  of  time-phased  work  elements  which  bring  the  product 
into  being.  The  point  of  making  this  distinction  is  that  the  program  is  basi¬ 
cally  a  set  of  R&D  activities,  and  encompasses  many  products  which  are  not 
parts  of  the  delivered  system.  Also,  the  complete  operational  system  may  involve 
elements  which  were  not  acquired  under  a  given  system  program;  for  example, 
neither  of  the  two  BUIC  systems  could  operate  without  the  radar,  communications, 
and  other  elements  which  were  acquired  as  parts  of  other  air  defense  programs. 

BUIC  II  and  BUIC  III  were  closely  related  not  only  as  systems  but  also  as 
system  programs.  BUIC  II  provided  the  direct  precedent  for  defining  the  scope 
of  contractor  responsibilities  associated  with  computer  program  development  in 
BUIC  III;  and  BUIC  II,  in  turn,  was  an  outgrowth  of  the  developmental  history 
of  SAGE.  Relevant  aspects  of  those  earlier  programs  are  therefore  reviewed 
briefly  in  the  following  sections  of  this  chapter,  to  provide  a  basis  for 
understanding  the  new  elements  which  were  introduced  in  BUIC  III. 

B .  SAGE 

As  the  first  big  computer-based  system  to  be  developed  at  the  Hanscom  complex 
(now  ESD) ,  SAGE  established  procedents  which  have  influenced  most,  if  not  all, 
of  the  later  L-System  programs.  Beginning  as  an  R&D  project  at  the  MIT  Lincoln 
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Laboratory  in  1951,  it  merged  into  the  Experimental  SAGE  Sector  (ESS)  in  1953, 
became  operational  at  its  first  site  (McGuire  AFB)  in  1958,  and  has  continued 
to  grow  and  change  to  the  present  time*  Thus,  its  history  covers  the  time 
span  during  which  the  data  processing  arts  have  had  their  growth,  essentially 
from  a  state  of  infancy,  to  their  current  and  rapidly  expanding  roles  of 
importance  as  elements  of  military  system  programs.  SAGE  was  directly  re¬ 
sponsible  for  much  of  the  growth;  in  the  late  1950s,  more  than  half  of  the 
nation’s  computer  programmers  were  still  associated  with  the  program. 

It  was  a  natural  result  of  its  close  association  with  the  evolving  state-of- 
the-art  in  data  processing  that  SAGE  was  adopted  as  a  "model"  for  command  and 
control  system  development.  Constant  changes  in  configuration  were  its  most 
distinguishing  characteristic.  Factors  leading  to  the  changes  included  the 
advent  of  new  weapons,  equipment,  and  tactical/strategic  doctrines,  as  well  as 
improved  computer  programming  techniques.  The  concept  of  ’’evolutionary  de¬ 
velopment"  was  derived  from  that  experience  and  generalized  as  a  guiding 
principle  for  command  and  control  system  programs.  This  principle  has  a  variety 
of  facets  which  are  closely  in  conflict  with  current  objectives  to  procure 
systems  as  items  having  pre-defined  performance,  schedules,  and  costs.  Hence, 
it  provided  some  of  the  basis  of  early  resistance  at  ESD  to  the  new  systems 
management  philosophy  (see  Chapter  II)  and  the  feasibility  of  developing 
computer  programs,  in  particular,  under  conditions  of  fixed  performance,  cost, 
and  schedule  requirements  has  continued  to  be  a  matter  of  debate. 

The  evolution  of  SAGE  computer  programs  occurred  in  a  variety  of  forms,  but 
notably  through  the  development  of  a  series  of  "SAGE  Models".  The  first 
operational  model,  which  was  installed  at  the  New  York  Air  Defense  Sector  at 
McGuire  AFB  in  1958,  was  based  on  the  Model  Zero  that  had  resulted  from  the 
experimental  developments  at  Lincoln  Laboratory.  Subsequent  models  incorpora¬ 
ted  various  computer  program  improvements,  but  were  mainly  in  response  to 
requirements  for  new  capabilities  or  to  changes  in  system/equipment  configura¬ 
tion  and  interfacing  weapons.  By  the  time  the  BUIC  II  program  was  initiated 
in  1962,  Model  9  was  operational  and  a  next  was  in  the  planning  stages.  The 
last  model  was  never  implemented,  although  new  "versions"  incorporating  exten¬ 
sive  changes  have  continued  to  appear  at  the  rate  of  3  or  4  per  year  since 
that  time. 

Each  new  model  or  version  represents  a  complete  re-cycling  of  the  computer  pro¬ 
gram  acquisition  process  through  its  6  phases  of  analysis,  design,  development, 
testing,  installation,  and  maintenance.  The  process  naturally  emphasizes  the 
added  or  altered  operational  capabilities,  but  it  also  typically  involves 
altering  other  parts  of  the  "basic"  computer  programs  to  avoid  exceeding  fixed 
storage  constraints  and  compensate  for  interactions.  The  analysis  phase  in¬ 
volves  conducting  studies  of  feasibility  and  trade-offs,  and  defining  firm 
requirements  to  be  incorporated  in  the  new  model  or  version.  The  design  phase 
consists  of  both  (1)  "operational  design",  which  results  in  the  precise 
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formulation  of  performance-level  requirements,  and  (2)  design  of  the  computer 
program  changes  to  meet  the  requirements.  Development  encompasses  the  coding, 
testing,  and  debugging  of  individual  computer  programs,  while  the  test  phase 
is  devoted  to  assembly  testing  of  functionally-related  elements,  checkout  of 
the  integrated  computer  program  system  at  an  operational  site,  and  the  per¬ 
formance  of  tests  under  live  conditions.  During  installation,  site-specific 
adaptation  data  are  incorporated  and  the  new  version  is  checked  for  operation 
at  each  site.  Once  operational,  a  maintenance*  phase  begins  and  continues  for 
the  life  of  that  model  or  version,  which  consists  of  (1)  malfunction  diagnosis 
and  error  correction  and  (2)  incorporating  minor  or  emergency  modifications 
on-site,  pending  appearance  of  the  next  complete  new  version. 

Involvement  in  SAGE  computer  programming  by  the  organizational  unit  of  RAND 
Corporation  which  eventually  became  SDC  was  an  outgrowth  of  earlier  work  in 
the  field  of  operational  readiness  training  for  personnel  of  the  manual  air 
defense  system.  The  capability  known  as  the  Manual  System  Training  Program 
(MSTP)  had  been  developed  and  put  into  operational  use  in  the  manual  air 
defense  system  during  the  period  of  1954  to  1957.  This  capability  consisted 
of  a  complex  set  of  simulation  equipment,  materials,  and  procedures  which  could 
be  used  at  the  operational  sites,  permitting  personnel  manning  the  air  defense 
network  to  conduct  realistic  exercises  against  simulated  air  attacks.  Following 
the  introduction  and  use  of  MSTP,  requirements  were  established  for  a  similar 
capability  to  be  provided  in  SAGE.  The  result  was  the  SAGE  System  Training 
Program  (SSTP),  which  was  developed  and  implemented  concurrently  with  other 
SAGE  elements.  And  like  the  other  elements,  SSTP  underwent  a  continuing  series 
of  evolutionary  refinements  and  changes. 

The  close  association  of  computer  programming  and  training  was  a  natural  result 
of  their  mutual  technical  concern  with  and  dependence  on  the  detailed  pro¬ 
cedures  involved  in  air  defense  operations.  Also,  in  SAGE,  special  computer 
programs  emerged  as  important  elements  of  the  on-site  SSTP  capability;  and 
still  others  were  developed  to  generate  the  synthetic  input  data,  aids,  and 
materials  for  distribution  to  the  sites. 

As  integral  parts  of  the  computer  program  and  training  development  activities, 
extensive  documentation  was  associated  with  the  SAGE  models  and  SSTP  problems. 
These  documents  included  reports  of  special  studies  and  analyses,  operational 
specifications  for  the  computer  programs,  positional  handbooks  for  the  SAGE 


*  Use  of  the  term  "maintenance"  for  these  activities  is  commonly  accepted, 
although  some  other  term  might  be  preferable  in  the  interest  of  avoiding 
confusion  with  equipment  usage;  for  example,  "modification"  is  closer  to  the 
point,  since  the  indicated  action  in  cases  of  malfunctions  or  other  deficien¬ 
cies  is  always  to  alter  the  original  configuration,  rather  than  to  restore 
or  maintain  it. 
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operational  personnel,  SAGE  System  Description,  adaptation  data,  and 
positional  guides,  manuals,  and  other  materials  for  SSTP. 

C.  BUIC  II 

The  conceptual  studies  leading  to  the  choice  of  a  system  configuration  for 
BUIC  II  were  carried  out  principally  during  1961.  The  document  which  served 
as  a  system  specification,  entitled  BUIC  System  Functional  Design  and  Perfor¬ 
mance  Requirements,  was  issued  as  ESD  Document  No.  416L-1  in  February  1962. 

There  followed  a  period  of  negotiations,  resulting  in  the  award  of  contracts 
to  two  associate  contractors  prior  to  mid-1962.  These  were:  Burroughs  Corpora¬ 
tion,  for  system  computer  and  associated  equipment;  and  System  Development 
Corporation,  for  computer  programs. 

Although  the  initiation  of  BUIC  II  thus  preceded  the  appearance  of  the  DoD 
Draft  Instruction  on  Program  Definition  by  about  a  year,  pressures  for  firm 
planning  of  costs  and  schedules  were  in  evidence  from  the  outset  of  the  program. 
The  equipment  contract  was  negotiated  on  a  fixed-price  basis;  and  the  equipment 
schedules  and  constraints  soon  became  the  pacing  factors  in  the  program  as  a 
whole.  At  that  time,  ESD  studies  which  led  to  the  MITRE  TM-3551  (see  Chapter 
II)  were  just  beginning,  and  little  or  no  guidance  was  available  to  the  SPO 
for  planning  the  management  of  computer  program  acquisition,  other  than  that 
which  could  be  supplied  by  MITRE  and  SDC  as  the  program  progressed. 

At  SDC,  SAGE  experience  provided  a  ready  basis  for  defining  the  major  technical 
products.  However,  in  SAGE,  those  products  had  come  into  being  gradually, 
through  the  "evolutionary"  processes  of  experimental  developmental  and  successive 
modifications  based  on  use  under  operational  conditions.  Such  factors  as  the 
brief  and  fixed  timing  of  acquisition,  concurrency  of  equipment  development, 
close  schedules,  and  the  general  austerity  of  funding  which  characterized  BUIC 
II  throughout  its  course  were  novelties.  Hence,  the  scope  and  content  of  work 
elements  for  the  computer  program  contractor  had  to  be  worked  out  for  the  first 
time  in  BUIC  II  under  conditions  which  were  significantly  closer  to  those  of 
current  system  programs  than  had  been  true  earlier.  The  resulting  elements 
are  outlined  briefly  below. 

Operational  Design 

As  it  has  been  defined  traditionally  in  SAGE  and  BUIC,  this  is  the  set  of 
activities  which  results  in  completed  "operational  specifications"  for  the 
computer  programs.  As  a  level  of  technical  work,  it  is  comparable  in  a 
general  way  to  the  system  engineering  effort  which  a  contractor  would  carry 
out  to  produce  Part  I  specifications  for  contract  end  items,  although  with 
some  discrepancies.  It  encompasses  such  component  elements  as:  analysis  of 
user  requirements;  study  of  alternative  solutions  and  trade-offs;  development 
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of  detailed  operating  descriptions  of  computer  functions,  controls,  and 
displays;  derivation  and  selection  of  mathematical  formulas  and  equations; 
specification  of  interconnections  between  console  switches  and  computer  (Vari¬ 
able  Display  Equipment);  and  writing  of  the  operational  specifications. 

Computer  Program  Design,  Development,  and  Test 

Although  not  known  as  contract  end  items  at  that  time,  elements  of  the  BUIC  II 
computer  program  system  were  separately  identified  as:  operational;  utility 
and  support  systems,  including  an  automatic  adaptation  capability;  Field  Site 
Production  System  (FSPS)  for  on-site  generation  of  Problem  Input  (PI)  tapes 
used  in  system  exercising;  and  the  BUIC  Data  Reduction  System  (BUDR)  for  a 
variety  of  uses  in  testing,  checkout,  system  exercising,  and  operational  data 
reduction.  The  development  process  for  each  of  these  elements  was  divided  into 
a  series  of  phases,  which  were  identified  in  one  BUIC  II  work  statement  as 
follows : 

a.  Program  Design:  allocation  of  functions  to  subprograms; 
allocation  of  drum  and  core  storage;  design  of  control 
and  timing  subprograms. 

b.  Program  Costing:  determination  of  machine  instructions, 
display  slots,  and  programming  personnel  requirements. 

c.  Subprogram  Design  and  Coding:  determination  of  logic, 
design  of  internal  and  external  communications,  and 
refinement  of  storage  allocation. 

d.  Simulation  Testing:  initially,  testing  of  subprograms 
(parameter  testing)  to  verify  conformance  with  subprogram 
design;  subsequently,  testing  at  successively  higher  levels 
of  assembly  to  verify  conformance  with  operational  specifi¬ 
cations,  using  simulated  inputs. 

e.  Documentation:  preparation  of  technical  documents  describing 
design,  data  structure,  testing  accomplished,  and  other 
information  for  using  and  modifying  the  computer  programs. 

f.  Retrofit:  incorporation  of  changes  to  meet  new  requirements. 

g.  Program  Maintenance  and  Error  Correction:  diagnosis  of 
malfunctions  and  correction  of  errors;  maintaining  cor¬ 
respondence  of  computer  programs  at  different  locations; 
maintaining  technical  documentation. 
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Program  System  Testing  (PST) 

Following  the  installation  of  system  equipment  at  an  operationally  configured 
facility,  the  computer  programs  resulting  from  in-plant  development  were 
installed  and  subjected  to  extensive  testing,  using  both  simulated  and  live 
inputs.  In  BUIC  II,  this  PST  activity  was  initiated  at  the  BUIC  Evaluation 
Facility  (BEF) ,  located  at  Hanscom  Field,  and  was  later  continued  at  the  first 
operational  site  at  North  Truro,  Massachusetts.  Category  II  tests  were  also 
accomplished,  to  some  degree  concurrently,  at  those  same  locations. 

Data  Collection  and  Processing 

This  activity  encompassed  the  collection  of  geographic,  unique-to-site ,  and 
weapons  characteristics  data,  as  well  as  processing,  verifying,  publishing, 
and  maintaining  the  data. 

Personnel  Subsystem  (PS) 


BUIC  II  was  the  first  system  program  at  ESD  to  be  materially  affected  by  the 
concept  of  the  personnel  subsystem,  which  had  become  established  as  Air  Force¬ 
wide  policy  through  the  issuance  of  AFL  375-5  ,  Planning  and  Programming' for 
System  Personnel,  in  October  1961.*  The  spectrum  of  PS  requirements  defined 
In  AFL  375-5  was  set  forth  in  the  BUIC  II  specification  issued  in  February 
1962,  although  stated  at  a  very  general  level.  They  posed  a  variety  of  problems 
which  were  new  in  the  ESD  system  context  and  which  were  also  somewhat  foreign 
to  the  PS  concept  itself  because  of  the  fact  that  PS,  like  other  systems 
management  elements  which  were  yet  to  appear,  had  originated  with  aircraft  and 
missiles.  There  ensued  a  period  of  study  to  define  implementing  methods  which 
would  be  suitable  to  the  requirements  of.  an  information  system,  and  to  identify 
workable  allocations  of  responsibilities  to  the  SPO,  MITRE,  and  the  two  associate 
contractors.  As  a  result,  BUIC  II  preceded  BUIC  III  as  a  “test  bed"  for  the 
requirements  in  this  area  as  they  relate  to  computer  program  acquisition. 

Elements  assigned  to  SDC  as  PS  contract  activities  are  summarized  briefly 
below.** 


a.  Personrel-Equipment  Data  (PED) .  PED  was  only  partially 
implemented  in  BUIC  II  because  of  the  general  austerity 
of  funding.  Information  was  compiled  and  maintained  by 
each  contractor  for  internal  purposes  only,  at  the  level 

of  whole  documents  rather  than  as  individual  data  elements. 

b.  Human  Engineering,  Detailed  task  analyses  for  operating 
system  personnel  were  required  as  a  basis  for  design  of 


*  The  Air  Force  letter  was  later  superseded  by  AFR  30-8. 

**  An  expanded  description  of  the  PS  adaptations  arrived  at  in  BUIC  II  is  con¬ 
tained  in  SDC  TM-1494,  The  Personnel  Subsystem  Program  for  an  Information 
System,  dated  23  September  1963. 
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display  format/content,  switch  functions,  and  special 
computer  program  functions,  and  to  provide  the  basic 
data  for  positional  handbooks  and  QQPRI. 

c.  Qualitative  and  Quantitative  Personnel  Requirements 
Information  (QQPRI) .  Position  description  information 
was  prepared  for  the  operator,  simulation/exercising, 
and  computer  programming  support  (Blue  Suit)  personnel, 
for  input  to  the  system  QQPRI  report  which  contained 
similar  information  for  equipment  maintenance  and  other 
personnel  supplied  by  the  hardware  contractor. 

d.  Personnel  Subsystem  Test  and  Evaluation  (PSTE) .  Con¬ 
tractor  support  was  provided  to  develop  plans  and  assist 
in  conducting  PSTE  during  Category  II  system  testing. 

e.  System  Exercising  for  Training  and  Evaluation  (SETE) . 
Although  not  included  as  a  recognized  element  of  the 
general  PS  structure,  SETE  was  so  classified  in  BUIC 
because  of  its  emphasis  on  training.  Development  of 
the  SETE  capability  in  BUIC  IT  was  based  on  the  SAGE 
model,  SSTP,  but  with  the  addition  of  requirements  per¬ 
taining  to  use  for  evaluating  system  functions  and  sub¬ 
functions  as  well  as  training  and  evaluating  personnel . 
Tasks  required  to  accomplish  the  development  included 
the  following  variety  of  elements: 

(1)  analyses  of  training  and  evaluation  requirements; 

(2)  specification  of  exercising  modes  and  configurations 

(3)  identification  of  performance/design  requirements 
for  special  equipment  for  problem  generation,  sim¬ 
ulation,  communications,  and  recording; 

(A)  specification  of  computer  programs  for  data 
reduction  and  recording; 

(3)  development  of  computer  programs  for  on-site 
generation  of  simulated  aircraft  tracks; 

(6)  development  of  problems  for  system-wide  exercises, 
together  with  associated  aids  and  materials; 

(7)  development  of  manuals  and  guides  covering  the 
organization,  planning,  and  conduct  of  system 
exercises . 
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Configuration  Management 


The  circumstances  under  which  configuration  management  was  introduced  in  BUIC 
II  were  mentioned  earlier,  in  Chapter  II  of  this  report.  At  the  time  the  con¬ 
figuration  management  exhibit  was  placed  on  contract  (June  1964),  the  documen¬ 
tation  to  be  controlled  either  was  already  completed  or  was  in  advanced  stages 
of  development.  Hence,  the  exhibit  merely  identified  the  specific  documents 
which  constituted  the  baselines,  in  terms  of  their  contractor  document  numbers, 
titles,  and  dates,  rather  than  specifying  their  format  and  content.  The  base¬ 
lines  identified  were  (1)  operational  specifications  which  had  been  written 
for  the  operational  field-site  problem  generation  (FSPS) ,  and  data  reduction 
(BUDR)  computer  programs,  and  (2)  a  series  of  user  manuals  which  was  in  process 
of  being  completed  for  the  utility  and  support  elements.  Also,  as  compared 
with  subsequent  exhibits,  there  was  no  attempt  to  identify  a  FACT  or  establish 
product  configuration  baselines.  In  other  respects,  the  control  and  document 
maintenance  procedures  were  closely  similar  to  those  which  were  later  adopted 
for  general  use,  although  with  a  number  of  minor  differences  in  the  names  and 
formats  of  control  and  maintenance  forms.  In  June  1964,  the  contract  identi¬ 
fied  the  following  specific  activities  under  a  task  entitled  "Configuration 
Management" : 

a.  Prepare  Change  Proposal  (CP)  forms  for  all  proposed  changes 
to  the  established  computer  program  baselines. 

b.  Prepare  Change  Recommendation  (CR)  forms  for  any  system  item, 
when  considered  necessary  because  of  conditions  affecting  the 
computer  program  developments. 

c.  Provide  summary  cost  estimates  for  each  Class  I  CP. 

d.  Publish,  maintain,  and  revise  the  Configuration  Index. 

e.  Publish  and  update  the  Configuration  Management  Status 
Report . 
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CHAPTER  V 


BUIC  III  PROGRAM  REQUIREMENTS  AND  EVENTS 


A.  INTRODUCTION 

The  Air  Force  was  authorized  by  the  Defense  Department  to  proceed  with  the 
BUIC  III  program  in  November  1964 .  This  event  had  been  preceded  by  a  variety 
of  studies  involving  ADC,  ESD,  MITRE,  SDC,  Rand,  and  others,  as  a  result  of 
which  the  BUIC  III  concept  had  been  selected  from  among  several  alternatives. 

As  a  fundamental  aspect  of  the  BUIC  III  concept,  the  program  was  initiated  as 
a  major  "modification"  of  the  BUIC  II  system,  rather  than  as  a  completely  new 
system  program,  and  prior  to  the  time  that  BUIC  II  was  scheduled  to  be  com¬ 
pletely  operational.  Thus,  it  readily  became  an  extended  and  modified  effort 
of  the  same  agencies,  including  the  same  principal  contractors  (Burroughs  and 
SDC)  who  had  been  involved  in  the  development  of  BUIC  II.  A  Definition  Phase 
was  not  required  since  estimated  costs  were  below  the  stated  DOD  thresholds. 
The  draft  system  specification  was  developed  during  the  first  half  of  1965  by 
ESD/MITRE,  with  some  support  by  the  contractors,  following  which  formal  con¬ 
tract  work  initiating  system  acquisition  began  at  the  outset  of  FY  1966  (July 
1965). 

The  BUIC  III  system  specification  was  the  first  to  be  produced  at  ESD  which 
attempted  to  conform  to  the  guidance  established  in  Exhibit  I  of  AFSCM  375-1, 
which  had  been  issued  in  essentially  its  present  form*  and  coordinated  among 
the  AFSC  divisions  during  the  preceding  year.  The  specification  defined 
performance/design  and  test  requirements  for  the  system,  including  require¬ 
ments  for  the  system  exercising  capability**, and  allocated  requirements  among 


*  The  January  1964  draft,  which  was  issued  for  official  use  in  June  1964,  was 
a  major  expansion  and  revision  of  an  earlier  version  of  AFSCM  375-1  which 
had  appeared  in  June  1962. 

**  The  label  carried  by  this  capability  in  both  BUIC  II  and  BUIC  III  is  System 
Exercising  for  Training  and  Evaluation  (SETE) .  Although  often  thought  of 
as  part  of  the  computer  program  system  segment  because  of  SDC’s  responsi¬ 
bility  for  its  extensive  computer  programming,  problem  generation,  and 
personnel  subsystem  aspects,  the  capability  is  more  properly  treated  —  as 
it  was  in  the  BUIC  III  System  Specification  —  as  a  system-level  set  of 
requirements.  Its  impact  on  the  system  includes,  for  example,  additional 
capacity  of  the  operational  computer,  as  well  as  special  consoles,  communi¬ 
cations,  simulation  equipment,  and  personnel. 
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the  five  system  segments: 


Data  Processing  and  Display  System  Segment 
Maintenance  Computer  Program  System  Segment 
Operational  Computer  Program  System  Segment 
Building  and  Facilities  System  Segment 
Communication  System  Segment 

Major  developmental  work  to  be  performed  by  the  two  associates  system  con¬ 
tractors  was  defined  by  the  first  three  of  these  segments.  The  equipment  and 
maintenance  computer  program  segments  were  assigned  to  Burroughs,  while 
activities  associated  with  the  operational  computer  program  were  assigned  to 
SDC . 

To  provide  a  basis  for  discussing  experience  with  the  systems  management  tech¬ 
niques,  the  following  two  sections  are  devoted  to  outlining  the  elements  of 
work  which  comprised  SDCfs  system  segment,  and  describing  the  ways  in  which 
these  were  influenced  by  the  new  management  procedures  and  requirements.  Sub¬ 
sequently,  in  the  final  section  of  this  chapter,  an  attempt  will  be  made  to 
present  a  factual  summary  of  events  as  they  actually  occurred  during  the  pro¬ 
gram. 


B.  THE  OPERATIONAL  COMPUTER  PROGRAM  SYSTEM  SEGMENT 

The  basic  items  to  be  developed  and  delivered  under  the  contract  were  three 
computer  program  contract  end  items  (CPCEIs),  which  were  identified  as: 

ADP  —  Air  Defense  Computer  Program 
SEP  —  System  Exercise  Computer  Program 
UCP  —  Utility  Computer  Program 

It  has  been  mentioned  earlier  (Chapter  IV)  that  these  correspond  closely,  with 
regard  to  general  computer  program  types  and  functions,  with  the  computer  pro¬ 
grams  which  had  been  required  in  both  SAGE  and  BUIC  II. 

In  addition  to  the  CPCEIs,  important  items  of  deliverable  data  included  the 
computer  program  specifications,  positional  handbooks,  user  manuals,  exercising 
manuals  and  guides,  and  system  exercise  problem  inputs  and  materials.  For  the 
most  part,  these  items  had  precedents  in  BUIC  II  (cf.  preceding  chapter). 

The  technical  activities  were  also  similar  to  those  which  had  been  carried  out 
under  the  BUIC  II  program,  except  for  the  pervasive  influence  of  BUIC  III 
management  innovations.  These  included:  operational  design;  computer  program 
design,  development,  and  testing;  development  of  system  exercising  problems, 
aids,  guides,  and  materials;  and  performance  of  associated  personnel  sub¬ 
system  tasks.  In  addition,  SDC  became  responsible  eventually  for  the 
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management  and  conduct  of  the  Category  II  test  program. 

However,  essentially  all  of  these  activities  were  affected  in  varying  degrees 
by  the  procedures  and  guidelines  which  had  been  developed  by  the  SDC  Systems 
Management  Project  during  the  first  six  months  of  FY  1965  (see  Chapter  II) . 

That  effort  had  been  funded  by  the  416M  SPO,  with  the  explicit  intent  to 
acquire  products  which  could  be  applied  and  tested  in  the  BUIC  III  program. 
Also,  the  SPO  was  actively  represented*  in  the  ESD  Software  Management 
Committee  which  reviewed  and  coordinated  the  effort  during  that  phase  of  the 
project.  As  a  result,  it  is  of  note  that  some  of  the  specific  new  requirements 
were  influenced  by  known  characteristics  of  the  BUIC  III  system.  This  was 
considered  to  be  a  desirable  circumstance  by  the  committee  members,  who 
regarded  the  development  of  generalized  L-System  procedures  to  be  the  primary 
objective,  but  felt  at  the  same  time  that  the  realism  of  concepts  would  be 
materially  enhanced  to  the  degree  that  they  could  be  related  to  the  parti¬ 
culars  of  an  actual  program.  Thus,  BUIC  III  became  a  "test  bed"  for  guiding 
the  initial  formulation  of  many  of  the  concepts,  as  well  as  for  subsequently 
evaluating  the  experience  with  their  application. 

The  procedures  applied  in  BUIC  III  consisted  of  adaptations  to  computer  pro¬ 
grams  of  established  Systems  Command  procedures,  principally  in  the  areas  of 
data  management  (AFSCM/AFLCM  310-1),  configuration  management  (AFSCM  375-1), 
and  testing  (AFSCM  375-4;  AFR  80-14).  Except  for  formal  design  reviews,  the 
program  was  not  specifically  influenced  by  the  concepts  and  requirements  of 
system  engineering  management  as  set  forth  in  AFSCM  375-5.  The  specific  doc¬ 
uments  which  governed  the  application  of  these  requirements  in  BUIC  III  were: 

(1)  AFSCM/AFLCM  310-1,  together  with  AFLC/AFSC  Forms  9  which  were  then  largely 
ESD-unique,  (2)  the  BUIC  III  configuration  management  exhibit,  ESD-416M-43** , 
and  (3)  the  contract  statement  of. work,  which  incorporated  certain  require¬ 
ments  in  the  computer  program  testing  area,  in  particular,  which  were  not 
covered  elsewhere.  During  the  first  year  of  this  contract,  a  few  of  the  Forms 
9  were  published  in  310-1,  and  such  other  documents  as  ESD  Exhibit  EST-2 
(later  superseded  by  ESD  Supplement  1  to  AFSCM  375-4)  and  ESDP  375-10  (sub¬ 
sequently  ESD  Exhibit  EST-3)  appeared. 

The  following  subsections  describe  the  requirements  which  were  introduced  in 
BUIC  III.  Most  of  these  were  novel,  in  the  sense  that  they  did  not  exist 


*  By  Lt.  B.  Capehart 

**  The  exhibit  was  based  upon  the  same  materials  which  were  issued  a  few  weeks 
later  as  ESD  Exhibit  EST-1.  Except  for  being  a  self-contained  document 
(whereas  EST-1  is  in  the  form  of  change  pages  to  AFSCM  375-1) ,  differences 
in  content  as  compared  with  EST-1  are  negligible. 
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in  the  same  form  in  BUIC  II.  For  convenience.  Table  5-1  presents  a  summary 
list  which  identifies  the  particular  aspects  of  novelty  as  compared  with 
earlier  practice. 

C.  SYSTEMS  MANAGEMENT  INNOVATIONS 
Data  Management 


The  use  of  a  Contractor  Data  Requirement  List  (CDRL:  DD  Form  1423)  was 
initiated  in  BUIC  III,  together  with  the  application  of  Forms  9  governing  the 
preparation  of  identified  data  items.  The  applicable  Forms  9  included  a  set 
which  had  been  developed  under  the  Systems  Management  Project  to*  cover  items 
which  were  judged  to  be  of  typical  interest  in  ESD  systems,  but  which  had  not 
previously  existed  in  AFSCM/AFLCM  310-1.  As  listed  below,  these  items  were 
initially  ESD-unique;  nearly  all  have  since  been  published  in  Volume  II  of 
310-1: 


Positional  Handbook — Information  System  Operational  Personnel 
(ESD  178)  (H-109*) 

Contract  End  Item  Detail  Specification  (Computer  Program) ,  Part 
I  (ESD  236)  (C-132*) 

Contract  End  Item  Detail  Specification  (Computer  Program)  Part  II 
(ESD  237)  (C-133*) 

Category  I  Test  Plan  (Computer  Program),  (ESD  261)  (T-103-1*) 

Category  I  Test  Procedures  (Computer  Program),  (ESD  262)  (T-103-1*) 

Category  I  Test  Report  (Computer  Program) ,  (ESD  263)  (T-118-1*) 

Exercise  Conduct  Manual  (ESD  281)  (Q-125-1*) 

Synthetic  Inputs  Operator  Guide  (ESD  282)  (Q-123-1*) 

Evaluation  Manual  (Information  System  Exercising  Personnel), 

(ESD  283)  (Q-124-1*) 

Human  Operator  Task  Analysis  for  Information  Systems 
(Q-] 5-18.0)  (Q-118-1*) 

Training  Needs/Exercise  Requirements  Analysis  (Q-26-18.0)  (Q-119-1*) 

*  Refers  to  the  data  item  number  (DD  Form  1664)  in  Volume  II  of  AFSCM/AFLCM 
310-1  as  of  30  August  1968. 
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Table  5-1.  Summary  of  Computer  Program  Management 
Requirements  Introduced  in  BUIC  III. 


ITEM 


PRECEDENTS 


AFSCM/AFLCM  310-1 
Requirements 


Some  Data  Items 


Part  I  Specification 


Ops  Specs 


H 

2 

W 


O 

< 


2 

O 


2 

o 

M 

2 

o 

u 


Part  II  Specification 


TM-1333 

User  Manuals  (U) 


Control 

& 

Accounting 


CP 

CP  (II) 

CR 

Conf.  Index 
Status  Rpt . 


Part 
►  Only 


FAC  I 


cn 

w 


a 


55 

o 


PDR 


Informal 


CDR 


cn 

w 

Q 


H 

cn 

w 

H 


& 

C 

o 

w 

< 


Part  I  Spec  (Sec  4) 
Cat  I  Test  Plan 
Cat  I  Test  Proc. 

Cat  I  Test  Reports 

PQTs 

FQT 

CPT&E 


u 


I P&A  Testing 
I  PST 


NOVEL  ELEMENTS 


Use  of  CDRL 

Specified  Format /Content 
Some  Items 


Quality  Ass.;  Interfaces 
Utility  Coverage 
Spec.  Conventions 


Comprehensive  Coverage 
Up-to-date  Accuracy 


ECP 

CR 

SCN 

SCL 

EICC 


Part  I 
& 

Part  II 


Conf.  Index 
Ch  St  Rpt 
V. D.  Doc 


Formal  Audit 


Reissue 


PCB 


Formal  Review 
Minutes 


Formal  Review 
Incremental  (w/PQTs) 
Minutes 


Required  (ESD  236) 
Required  (ESD  261) 
Required  (ESD  262) 
Required  (ESD  263) 
Required 
Required 


(no  change) 
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Evaluation  Needs/Exercise  Requirements  Analysis 
(Q-24-18.0)  (Q-117-1*) 

Exercising  Capability  Implementation  Plan 
(Q— 27 — 29 . 0)  (Q-120-1*) 

Minutes  of  Formal  Reviews  and  Inspections 
(ESD  289)  (C-131*) 

Users  Manual  (Computer  Program) ,  (ESD  290)  (H-110*) 

Version  Description  Document  (Computer  Program) 

(ESD  291)  (C-135*) 

Configuration  Index  (Computer  Program) 

(ESD  293)  (C-136*) 

Change  Status  Report  (Computer  Program) 

(ESD  293)  (C-137*) 

System  Exercising  Problem  Package 
(Q-28-56 . 0)  (Q-121-1*) 

System  Exercising  Problem  Agreements  Document 
(ESD  295) 

It  was  indicated  earlier  that  most  of  the  above  items,  as  well  as  others 
listed  in  the  CDRL  which  were  not  unique,  had  precedents  among  deliverable 
items  of  data  which  had  been  prepared  in  BUIC  II.  However,  the  CDRL  pro¬ 
cedures  as  such,  a  few  of  the  data  items  (notably,  in  the  test  area),  and  a 
number  of  specific  format/content  requirements  were  introduced  for  the  first 
time  in  BUIC  III. 

Specifications 

The  Forms  9  for  CPCEI  Detail  Specifications,  both  Parts  I  and  II,  were  derived 
from  studies  which  had  attempted  to  take  into  account  a  complex  of  considera¬ 
tions  posed  by  (1)  the  general  characteristics  of  uniform  specifications  as 
embodied,  e.g.,  in  Exhibit  II  of  AFSCM  375-1,  (2)  previous  practices  in  doc¬ 
umenting  computer  program  developments,  as  exemplified  in  such  systems  as 
416L,  416M,  465L  and  473L,  and  (3)  the  similarities  and  differences  between 
computer  programs  and  equipment  which  relate  to  purposes  of  configuration 
management.  Hence,  they  involved  a  number  of  novel  elements  in  BUIC  III,  as 


*  Refers  to  the  data  item  number  (DD  Form  1664)  in  Volume  II  of  AFSCM/AFLCM 
310-1  as  of  30  August  1968. 
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compared  with  the  "specifications"  which  had  been  written  previously  for 
BUIC  II.  The  major  differences  may  be  summarized  as  follows: 


1.  The  Part  I  Specification  is  basically  a  performance-level 
specification,  as  were  the  earlier  "operational  specifications 
(ops  specs)"  which  had  been  in  use  for  SAGE  and  BUIC  II.  How¬ 
ever  : 

a.  Ops  specs  had  previously  been  written  and  provided  to 
the  user  only  for  operational  and  support,  but  not  for 
utility,  computer  programs.  Hence,  the  requirements 
to  write  a  performance-level  specification  for  utility 
elements  (including  the  compiler)  was  completely  new. 

b.  The  ops  specs  had  not  contained  a  quality  assurance 
section  comparable  to  Section  4  of  the  Part  I 
specification. 

c.  Interfaces  and,  in  general,  the  specific  identification 
of  detailed  inputs  and  outputs  for  each  functional 
element  of  the  CPCEI,  had  not  been  systematically  in¬ 
cluded  in  ops  specs.  Equipment  interfaces,  as  well  as 
personnel  operating  procedures,  had  been  included  in  the 
ops  specs,  however,  often  in  the  form  of  detailed 
narrative  descriptions  of  system  operational  sequences 
and  conditions.  In  addition,  a  separate  set  of  documents 
known  as  "Variable  Display  Equipment  Specifications  (VDE 
specs)"  had  been  written,  defining  functions  and  labels 
of  console  switches  and  detailing  the  recommended  wiring 
of  console  switches  to  computer  storage  locations. 

2.  Essentially  all  of  the  computer  program  documentation  requirements 
tained  in  the  Form  9  for  the  Part  II  specifications  were  based 
upon  elements  which  had  previously  existed  in  some  form,  and 

for  some  computer  programs.  However,  the  requirement  had  not 
previously  existed  to  provide  all  of  those  elements  for  each 
CPCEI,  and  at  one  point  in  time.  Some  of  the  elements,  e.g., 
flow  charts,  had  been  prepared  at  early  phases  of  the  develop¬ 
ment  process,  but  had  not  been  prepared  at  detailed  levels  for 
all  components  of  each  computer  program.  Nor  had  the  design 
description  material  generated  during  developmental  phases, 
including  flow  charts,  been  revised  and  updated  on  a  con¬ 
tinuing  basis  to  reflect  actual  design  decisions  resulting 
from  the  processes  of  coding,  testing,  and  ongoing  changes. 

Thus,  the  requirement  to  prepare  the  type  of  comprehensive 
description  described  in  the  Form  9  for  a  Part  II  CPCEI 
specification,  which  would  be  accurate  in  all  respects  at 


con- 
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the  time  of  FACI  for  each  CPCEI,  represented  a  major  new  element 
in  the  BUIC  III  effort. 

Configuration  Management 

Most  of  the  procedures  and  standard  forms  used  for  computer  program  configura¬ 
tion  control  and  accounting  had  been  developed  during  the  period  of  1963-1964, 
and  had  been  used  in  BUIC  II.  Only  a  few  changes  of  a  minor  nature  were  made 
in  the  forms  used  in  BUIC  III,  which  conformed  in  all  essential  respects  to 
those  currently  described  in  ESD  Exhibit  EST-1.  These  were:  Engineering 
Change  Proposal;  a  Change  Report  form  for  reporting  Class  II  changes;  Speci¬ 
fication  Change  Notice;  Specification  Change  Log;  End  Item  Configuration  Chart; 
Configuration  Index;  Change  Status  Report;  and  Version  Description  Document. 

In  BUIC  II,  traditional  operational  specifications  (ops  specs)  comprised  the 
baselines,  which  were  established  and  formally  maintained  only  at  that 
level  —  i.e.,  of  performance,  rather  than  at  the  level  of  detail  design. 

In  fact,  the  controls  adopted  in  BUIC  II  were  largely  based  on  controls  which 
had  been  exercised  by  ADC  to  govern  performance-level  changes  in  the  SAGE 
system  for  some  years  previously.*  Format  and  content  requirements  for  the 
ops  specs  had  not  been  contractually  specified.  These  were  replaced  in  BUIC 
III  by  the  Part  I  specifications,  with  new  format/content  elements  as  described 
above,  which  became  approved  and  controlled  as  Design  Requirements  Baseline 
documents.  In  addition,  the  Part  II  specifications  were  also  required  as 
deliverable  items  under  the  contract,  and  were  scheduled  to  be  established  as 
Product  Configuration  Baselines  as  a  result  of  First  Article  Configuration 
Inspection  (FACI) . 

Thus,  while  configuration  control  at  the  performance  level  had  been  introduced 
earlier,  configuration  management  procedures  were  expanded  in  BUIC  III  to 
encompass  uniform  computer  program  specifications,  FACI,  and  control  of  base¬ 
lines  at  the  product  configuration  level. 

Design  Reviews 

The  Preliminary  Design  Review  (PDR)  and  the  Critical  Design  Review  (CDR) 
involved  formal  meetings  and  reviews  of  computer  program  design  data  which  had 
no  direct  precedents  in  earlier  systems.  Although  specific  design  questions 
had  occasionally  been  matters  of  interest  to  the  SPOfs  GSE/TDC,  MITRE,  during 


*  ADCM  55-32,  Processing  Adaptation  Data,  Computer  Program,  and  Related  Equip¬ 
ment  Changes  in  SAGE,  which  has  been  updated  a  number  of  times  since,  was 
issued  as  early  as  January  1961. 
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BUIC  II,  the  ESD  SPOs  had  not,  in  general,  been  able  to  acquire  and  maintain 
the  capability  to  monitor  contractor  computer  programming  efforts  at  the 
technical  design  level.  At  SDC,  the  entire  processes  of  computer  program 
design,  coding,  and  testing  had  been  carried  out,  of  necessity,  as  "in-house" 
contractor  responsibilities.  Thus,  the  objectives,  functions,  and  conduct  of 
PDRs  and  CDRs  became  matters  with  which  both  the  SPO  and  contractor  personnel 
gained  their  first  actual  experience  in  BUIC  III.  The  PDR  was  scheduled  as 
a  single  review  of  each  CPCEI  (with  minor  variations  noted  in  a  later  section 
of  this  chapter),  concerned  with  the  design  approach  to  the  CPCEI  as  a  whole. 
The  CDR  for  each  CPCEI  was  planned  to  be  carried  out  in  increments,  con¬ 
currently  with  Preliminary  Qualification  Tests  (PQTs) ,  as  a  review  of  the 
completed  design  of  CPCEI  components  undergoing  PQT. 

Note:  ESD  Exhibits  EST-1  and  EST-3  emphasize  CDR 

as  a  review  to  be  conducted  at  the  time  when  de¬ 
tailed  flow  charts  are  available  for  the  computer 
program  components,  prior  to  the  start  of  coding. 

However,  EST-1  also  allows  for  the  variation  chosen 
for  BUIC  III,  in  which  the  review  was  conducted  after 
coding  of  the  component  undergoing  review  was  com¬ 
pleted  . 

Category  I  Testing 

Prior  to  BUIC  III,  computer  program  testing  associated  with  the  earlier  air 
defense  systems  had  been  accomplished  by  the  contractor  without  formal  involve¬ 
ment  by  the  SPO.  The  test  process  included  in-plant  testing  of  components  as 
a  normal  adjunct  to  coding  and  assembly  operations  at  SDC.  In  addition,  it 
also  involved  the  conduct  of  a  comprehensive  checkout  of  the  computer  program 
at  the  Category  II  test  site  following  installation  of  equipment  and  computer 
programs  and  prior  to  the  formal  beginning  of  Category  II  tests.  This  check¬ 
out  operation  was  known  as  "Program  System  Testing  (PST)".  It  is  considered 
to  be  an  important  and  necessary  extension  of  the  development  process  for 
operational  computer  programs,  in  particular,  since  the  Category  II  site 
normally  provides  the  first  opportunity  to  exercise  the  complete  set  of  com¬ 
puter  programs  in  the  operational  equipment  configuration. 

The  Category  I  test  program  formulated  for  BUIC  III  consisted  of  the  elements 
which  are  now  described  elsewhere,  in  varying  degrees  of  detail,  in  such  doc¬ 
uments  as  ESD  Exhibit  EST-1,  ESD  Supplement  1  to  AFSCM  375-4,  SDC  TMs  3361  and 
3596,  ESD-TR-68-1,  and  associated  Forms  9  in  the  Test  Category  of  AFSCM/ AFLCM 
310-1,  Vol.  II.  The  major  elements  are  summarized  briefly  below: 

1.  Category  I  Test  Plan.  A  test  plan  was  required  for  each  of  the 
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three  CPCEIs.  These  plans  were  based  on  the  Form  9  (ESD  261), 
but  were  also  influenced  by  a  variety  of  specific  circumstances 
and  agreements  reached  with  the  SPO,  relating  to  both  structure 
and  content. 

2.  Classes  of  Testing.  By  analogy  with  the  types  of  Category  I 
testing  which  have  been  established  for  equipment  CEIs,  re¬ 
quirements  were  formulated  under  the  following  three  categories 

a.  Computer  Program  Test  and  Evaluation  (CPT&E) .  In 
general,  CPT&E  encompasses  the  testing  which  a  con¬ 
tractor  normally  conducts  as  an  integral  part  of  his 
design  and  development  effort.  Thus,  this  category 
subsumes  essentially  all  of  testing  which  had  been 
typical  of  the  preceding  systems,  as  described  above. 

Major  requirements  formally  recognized  by  the  SPO 
with  respect  to  CPT&E  pertain  to  in-plant  use  of 

GFE  computing  equipment,  and  to  utilization  of  the 
Category  II  test  site  for  computer  program  install¬ 
ation  and  checkout  (formerly  PST;  cf.  above). 

b.  Preliminary  Qualification  Test  (PQT) .  PQTs  are  con¬ 
ceived  as  interim  tests  which  are  intended  to  serve 
the  primary  purpose  of  demonstrating  contractor  pro¬ 
gress  towards  meeting  design  objectives  for  a  computer 
program  CEI.  As  expressed  in  current  guidance  documents 
(e.g.,  ESD  Exhibit  EST-1),  they  are  oriented  towards 
verifying  portions  of  the  CPCEI  prior  to  formal  quali¬ 
fication,  but  are  not  expected  to  accomplish  the  quali¬ 
fication  as  such.  This  concept  was  introduced  for  the 
first  time  in  BUIC  III,  without  benefit  of  either 
amplifying  guidance  or  experience  on  the  part  of  the 
SPO  or  contractor.  As  a  result,  PQTs  were  required 

and  performed  in  BUIC  III  with  somewhat  greater  emphasis 
(and  difficulty)  than  might  normally  be  expected.  PQTs 
were  scheduled  to  test /demonstrate  essentially  all  re¬ 
actions  within  each  function  of  the  ADP  and  SEP  CEIs, 
prior  to  formal  qualification  tests.  Each  PQT  was 
combined  with  an  incremental  CDR  of  design  data  per¬ 
taining  to  the  tested  components.  No  FQT  (Formal 
Qualification  Test)  and  PQTs  only  for  two  of  its 
many  functions  were  conducted  on  the  utility  com¬ 
puter  program  (UCP) .  To  save  time  and  cost,  com¬ 
plete  Category  I  testing  was  not  required  for  UCP, 
which  could  reasonably  be  considered  qualified 
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through  its  extensive  use  during  the  program  in 
developing  ADP  and  SEP. 

c.  Formal  Qualification  Test  (FQT) .  FQTs  were  required 
for  the  ADP  and  SEP,  and  were  scheduled  for  perfor¬ 
mance  at  the  Category  II  test  site  following  computer 
program  installation  and  checkout,  prior  to  FACI  and 
the  beginning  of  Category  II  testing.  Successful  com¬ 
pletion  of  FQT  was  planned  as  the  primary  testing  basis 
for  acceptance  of  the  CPCEI. 

3.  Test  Procedures  and  Reports.  Procedures  and  reports  were  required 

for  both  PQTs  and  FQTs,  in  conformance  with  ESD  262  and  263  (currently 
contained  in  Vol.  II  of  AFSCM/AFLCM  310-1  as  T-103  and  T-118) .  Test 
Procedures  documents  for  PQTs  were  required  for  SPO  review  30  days  in 
advance  of  scheduled  PQTs.  In  the  case  of  FQTs,  the  Test  Procedures 
document  was  required  90  days  in  advance,  for  SPO  review  and  approval 
and  to  accomplish  necessary  changes. 

In  summary,  the  Category  I  test  program  applied  to  BUIC  III  CPCEIs,  other  than 
utility,  covered  the  full  range  of  documentation  and  formal  testing  procedures 
which  have  since  become  incorporated  in  general  guidance  for  375-series  appli¬ 
cations  to  computer  programs.  The  novel  elements,  as  compared  with  previous 
SDC/416M  SPO  experience,  were  the  preliminary  and  formal  qualification  tests 
and  their  associated  test  plans,  procedures  and  reports. 

D.  PROGRAM  MILESTONES 

1 .  General 

A  summary  of  major  events  as  they  actually  occurred  during  the  program  is  pre¬ 
sented  graphically  in  Figure  5-1,  which  portrays  significant  dates  relating  to 
specifications,  design  reviews  and  FACI,  test  plans,  and  tests,  principally 
for  the  three  computer  program  contract  end  items  (CPCEIs)  in  question.  Dates 
on  which  the  system  specification  was  issued  and  the  Category  II  testing  was 
conducted  are  also  included,  however,  for  orientation  to  the  system  program  as 
a  whole. 

To  distinguish  among  the  three  CPCEIs,  different  symbols  are  used  for  the  Air 
Defense  Program  (ADP),  the  System  Exercise  Program  (SEP),  and  the  Utility  Com¬ 
puter  Program  (UCP) ;  these  are  triangles,  circles,  and  squares,  respectively. 
Symbols  are  drawn  as  solids  for  actual  past  events,  and  in  outline  only  for 
scheduled  future  events  which  had  not  occurred  at  the  time  the  chart  was 
drawn  (February  1969).  As  stated,  the  chart  represents  the  timing  of  actual 
occurrence,  prior  to  February  1969.  It  is  not  based  on  the  initial  program 
schedule,  nor  on  periodic  revisions  which  occurred  as  the  program  progressed. 
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Some  of  the  events  are  depicted  in  the  figure  in  a  somewhat  oversimplified 
manner,  in  the  interests  of  avoiding  clutter.  However,  more  precise  data, 
together  with  some  amplification  of  their  meaning  in  terms  of  related  events 
and  circumstances,  are  provided  in  the  narrative  below.  In  this  section, 
emphasis  is  placed  on  descriptive  and  factual  data  regarding  the  program  events 
Discussions  of  problem  areas  and  evaluative  comments  will  be  reserved  for  the 
following  chapter. 

2 .  System  Specification 

The  system  specification  for  BUIC  III  was  initially  issued  as  a  formal  document 
designated  SS-ES416M-65-1A,  on  1  June  1965.  It  had  been  developed  during  the 
first  six  months  of  1965  by  the  MURE  Corporation,  principally,  but  with  signi¬ 
ficant  inputs  from  the  two  associate  contractors,  SDC  and  Burroughs  Corporation 

A  major  revision,  designated  SS-ES416M-65-1B,  was  issued  exactly  one  year 
later,  on  1  June  1966.  The  revision  contained  significant  changes  and  expan¬ 
sions  resulting  from  contractor  efforts  in  developing  Part  I  specifications 
for  the  system  contract  end  items. 

These  two  events  are  depicted  by  star  symbols  in  the  top  row  of  Figure  5-1. 

3 .  Part  I  Specifications 

The  dates  of  Part  I  specifications  which  are  shown  in  Figure  5-1  for  the  three 
computer  program  CEIs  represent,  in  general,  the  principal  dates  of  basic 
issue  —  i.e.,  of  the  first  issues  in  accepted  forms  which  constituted  initial 
baselines  for  the  items.  In  all  cases,  basic  issues  were  preceded  by  one  or 
more  drafts  for  SPO  review  and  approval.  Hence,  except  for  formally  pro¬ 
cessed  subsequent  changes,  the  dates  attempt  to  represent  times  at  which  the 
Part  I  specification  developments  were  completed.  Certain  deviations  and 
other  amplifying  information  are  noted  below,  separately  for  each  of  the 
CPCEIs . 


AIR  DEFENSE  PROGRAM  (ADP) 

Because  of  its  complexity  and  bulk  (in  excess  of  2000  pages,  total)  the  ADP 
Part  I  specification  was  structured  into  a  set  of  11  volumes,  corresponding 
generally  to  a  breakdown  of  the  ADP  into  major  functional  groupings.  This 
multi-volume  structure  was  also  adopted  for  roost  of  the  other  principal  com¬ 
puter  program  documents,  as  will  be  explained  subsequently  for  each. 

The  date  indicated  in  Figure  5-1  actually  represents  completion  of  only  the 
first  10  of  these  volumes;  the  11th,  ,fMessage  Formats”,  was  delayed  for  an 
additional  six  months  by  the  need  to  resolve  interfaces  with  other  systems 
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(SAGE,  BUIC  II).  A  list  of  actual  dates  for  all  volumes,  together  with  their 
numbers,  titles,  and  functional  elements  covered  under  each,  is  presented  in 
Table  5-2.  For  convenience,  the  table  combines  the  Part  I  specification  data 
with  similar  data  relating  to  Category  I  Test  Plans,  which  will  be  discussed 
in  a  later  subsection. 


SYSTEM  EXERCISE  PROGRAM  (SEP) 

The  Part  I  specification  for  SEP  was  written  in  four  volumes.  The  first  two, 
"General"  and  ,fBUIC  Exercise  Preparation  System  (BEPS),n  were  issued  on  21 
January  1966  and  the  second  two,  "BUIC  Analysis  and  Reduction  System  (BARS)" 
and  a  "Classified  Supplement,"  were  issued  on  1  September *  1966 .  Table  5-3 
lists  these  volumes,  their  titles,  the  functional  elements  comprising  indiv¬ 
idual  chapters  under  each,  and  the  dates  of  basic  issue.  As  in  the  case  of 
ADP  above,  the  table  also  includes  Category  I  Test  Plan  data  which  will  be 
discussed  separately . 

The  widely-separated  dates  of  their  Part  I  specification  volumes  reflects,  in 
part,  the  fact  that  BEPS  and  BARS  were  relatively  independent  part  of  the  SEP. 

A  similar  separation  with  respect  to  major  milestones  tended  to  be  character¬ 
istic  throughout  the  program.  However,  the  late  initial  dates  of  BhRS  Part  I 
specifications  were  largely  a  result  of  new  requirements  associated  with  the 
System  Test  Reduction  Processor,  the  Subsystem  Test  Processor,  and  modification 
to  the  Test  Data  Analysis  Processor.  The  new  requirements,  which  were  introduced 
shortly  prior  to  the  original  completion  date,  added  about  40%  to  the  size  and 
occasioned  a  realignment  of  contractor  manpower  and  schedule. 

UTILITY  COMPUTER  PROGRAM  (UCP) 

The  UCP  Part  I  specification  consisted  of  four  volumes,  all  of  which  were 
issued  on  10  January  1966  as  indicated  in  Figure  5-1,  three  weeks  earlier 
than  the  ADP  and  SEP  specifications.  The  left-hand  column  of  Table  5-4  shows 
the  volume  numbers  and  titles  of  the  UCP  Part  I  specification.  The  second 
column  in  the  table  lists  the  functional  elements  which  comprise  individual 
chapters  in  the  various  volumes.  Note,  as  indicated  in  Table  5-4,  that  one 
function,  "Timing,"  was  added  to  Volume  3  on  5  November  1968.  This  addition 
was  the  result  of  ECP  120-2,  "Addition  of  a  Timing  Program  to  ADP/UCP,"  which 
provided  for  a  timing  tool  to  facilitate  estimating  the  operating  time  of  ADP 
programs  or  portions  thereof.  The  third  column  in  the  table  shows  the  dates 
of  issue  of  the  UCP  volumes  and  the  Timing  function  document.  The  last  two 
columns  contain  data  pertaining  to  Category  I  Test  Plans,  which  are  discussed 
in  the  following  section. 

4 .  Category  I  Test  Plan 

The  Category  I  Test  Plans  for  BUIC  III  CPCEIs,  like  the  Part  I  specifications, 
were  also  structured  into  volumes,  corresponding  roughly  to  functional  elements 
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Table  5-2.  Volume  Structure  of  Part  I  Specification  and  Category  I 
Test  Plan  for  the  Air  Defense  Program  (ADP) 


Part  I  Specification 

Cat. 

I  Plan 

Vol , 

Title 

Functional  Elements 

Dates 

"  Vol, 

Dates 

1 

General 

31 

Jan 

66 

1 

1  May  66 

2 

Air  Surveillance 

Radar  Inputs 

31 

Jan 

66 

2  . 

1  May  66 

Active  Tracking 

3 

1  May  66 

Passive  Tracking 

4 

28  Jan  66 

Height 

5 

21  Jan  66 

Positive  Target  Control 

6 

1  May  66 

Identification 

7 

1  May  66 

3 

Weapons 

Weapons  Assignment 
and  Commitment 

31 

Jan 

66 

8 

25  Jan  66 

Weapons  Direction/ 

9 

25  Jan  66 

Communication 

Air  Defense  Artillery 

10 

25  Jan  66 

4 

Information 
Transfer  and 

Manual  Inputs 

Information  Transfer 

31 

Jan 

66 

11 

24  Jan  66 

Manual  Inputs 

12 

1  May  66 

5 

St art over , 

Control,  Record¬ 
ing,  and  Real- 
Time  Simulation 

Startover 

31 

Jan 

66 

13 

1  May  66 

Control 

14 

1  May  66 

Recording 

15 

1  May  66 

Real-Time  Simulation 

16 

1  May  66 

6 

Adaptation 

31 

Jan 

66 

7 

Variable  Display 
Equipment 

31 

Jan 

66 

8 

Displays 

Display  Specification 
Situation  Displays 
Tabular  Displays 
Abbreviations  and 

31 

Jan 

66 

17 

1  May  66 

Definitions 

9 

Switch  Actions 

31 

Jan 

66 

JO 

Classified  Suppleme 

nt 

31 

Jan 

66 

11 

Message  Formats 

1  Aug  66 

- -  1 1 1. ■■■■■■  ii 
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Table  5^3.  Volume  Structure  of  Part  I  Specification  and  Category  I 
Test  Plan  for  the  System  Exercise  Program  (SEP) 


Part 

I  Specification 

Cat. 

I  Plan 

Vol. 

Title 

Functional  Elements 

Dates 

Vol. 

Dates 

1 

General 

31  Jan. 

66 

1 

Control 

Data  Base  Maintenance 
Octal  Correction 

2 

Exercise  Pre¬ 

paration  (BEPS) 

31  Jan. 

66 

1 

1  May  66 

Exercise  Tape 
Generation 

1 

Exercise  Tape 
Description 

1 

Exercise  Tape 
Modification 

1 

3 

Analysis  and 
Reduction 

System  (BARS) 

Exercise  Processor 

1 

Sept . 

66 

2 

15  Nov. 66 

Operational  Processor 

3 

15  Nov. 66 

Test  Data  Analysis 
Processor 

4 

15  Nov. 66 

System  Test  Reduction 
Processor 

5 

15  Nov. 66 

Subsystem  Test  Pro¬ 

6 

15  Nov. 66 

cessor 

4 

Classified 

Supplement 

1 

Sept . 

66 
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Table  5-4*  Volume  Structure  of  Part  I  Specification  and  Category  I 
Test  Plan  for  the  Utility  Computer  Program  (UCP) 


Part 

I  Specification 

Cat. 

I  Plan 

Vol. 

Title 

Functional  Elements 

Dates 

Vol. 

Dates 

1 

General 

10  Jan.  66 

2 

Assembly  Analysis 

Machine  Language 
Assembler 

JOVIAL  Compiler 

Assemble  Compool 

Assemble  Geography 

Binary  Data  Insertion 
Tape  Load 

Set/Use 

10  Jan.  66 

1 

1  May  66 

Indirect  Address  and 

BAR  Table  Reference 
Adaptation  Calculation 
Parameter  Test  Tool 

2 

1  May  66 

3 

Facility  System 

Facility  Control 
Pre-Recording 

Dynamite 

Symbolic  Relative 
Corrector 

10  Jan.  66 

Timing 

(added  5 
Nov.  68) 

4 

General  Utility 

Tape  File  Maintenance 
Binary  Read 

Load  Octals  on  Tape 

Tag  Reference 

Symbolic  Corrector 

Loader 

Input /Output 

Computer  Utility  and 
Support  System  Executive 
Dump  Function 

10  Jan.  66 
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of  the  computer  program.  In  general,  basic  issues  of  test  plan  volumes  were 
preceded  by  drafts  for  SPO  review  and  approval.  While  they  are  not  directly 
subject  to  configuration  management,  the  plans  were  also  maintained  on  a 
continuing  basis  during  Acquisition  to  reflect  approved  changes  to  the  Part  I 
specifications.  Accordingly,  they  were  frequently  updated  by  means  of  change 
pages  or  revision,  following  their  dates  of  basic  issue. 

In  Figure  5-1,  symbols  are  located  at  points  which  represent  the  basic  issue 
dates  for  groups  of  volumes.  Additional  explanation  is  provided  below  for 
each  of  the  three  computer  program  items. 

AIR  DEFENSE  PROGRAM  (ADP) 

The  Category  I  Test  Plan  for  ADP  was  written  in  a  total  of  17  volumes.  One 
of  these  (Vol .  1,  General)  emphasized  planning  of  formal  qualification  (FQT) 
for  the  CPCEI  as  a  whole.  The  other  16  were  devoted  individually  to  detailed 
planning  of  PQTs  for  the  computer  program  functional  elements  identified  in 
various  volumes  of  the  Part  I  specification,  primarily,  but  also  included 
planning  pertinent  to  FQT. 

Symbols  shown  in  Figure  5-1  are  located  at  two  different  dates  for  ADP, 
indicating  that  6  volumes  were  issued  in  January  and  the  remaining  11  volumes 
in  May  of  1966.  The  volume  numbers,  titles  (corresponding  to  the  Part  I 
specification  titles  shown),  and  dates  of  issue  are  listed  in  the  preceding 
Table  5-2.  It  may  be  noted  that  the  table  shows  an  absence  of  test  plan 
volumes  corresponding  to  switch  actions  and  message  formats.  These  were 
necessarily  included  as  essential  aspects  in  the  testing  of  other  functional 
elements . 


SYSTEM  EXERCISE  PROGRAM  (SEP) 

The  two  points  shown  in  Figure  5-1  for  the  SEP  Category  I  Test  Plan  also 
indicate  two  separate  dates  of  issue  for  the  test  plan  volumes,  in  May  and 
November  1966.  The  volumes,  dates,  and  correspondence  of  test  plan  breakdown 
with  elements  of  the  Part  I  specification  are  identified  in  the  preceding 
Table  5-3.  The  discrepancy  of  6  1/2  months  in  publication  dates  reflects, 
basically,  the  lag  in  BARS  portions  of  SEP  which  had  been  initiated  by  the 
earlier  delay  in  completing  BARS  volumes  of  the  Part  I  specification. 

UTILITY  COMPUTER  PROGRAM  (UCP) 

An  early  decision  had  been  reached  that  utility  computer  programs,  except  for 
two  designated  elements,  would  be  qualified  through  their  use  in  developing  the 
ADP  and  SEP.  As  a  result,  the  test  plan  for  UCP  was  confined  to  one  volume 
each  for  the  two  designated  elements,  the  JOVIAL  Compiler  and  BUIC  Adaptation 
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Calculation  (BAC) .  These  are  listed  in  the  preceding  Table  5-4.  In  Figure 
5-1,  the  symbol  represents  the  date  on  which  both  volumes  were  published. 

5 .  Preliminary  Design  Reviews  (PDRs) 

PDRs  for  the  three  CPCEIs  were  held  on  the  following  dates:  for  ADP,  on  14-15 
February  1966;  for  parts  of  SEP,  on  15-16  February  1966;  for  UCP  on  16 
February  1966;  and  for  remaining  parts  of  SEP,  on  23  August  1966.  The  two 
increments  of  PDR  for  SEP  basically  followed  the  discrepancy  between  BEPS  and 
BARS  which  had  been  generated  earlier  by  the  delay  in  the  BARS  Part  I  specifi¬ 
cation.  However,  the  February  PDR,  for  BEPS,  also  covered  the  BARS  Test  Data 
Analysis  Processor.  The  later  increment  was  devoted  to  the  BARS  Exercise 
Processor,  Operational  Processor,  System  Test  Reduction  Processor,  and  Sub¬ 
system  Test  Processor. 

The  review  in  each  case  was  concerned  with  preliminary  design  of  the  CPCEI  as 
a  whole,  with  respect  to  levels  of  design  information  which  would  subsequently 
comprise  a  general  volume  (Vol.  1)  of  each  Part  II  specification.  These 
included  attention  to  interfaces,  allocation  of  functions  to  computer  program 
components  (CPCs) ,  control  and  sequencing  of  CPC  operations,  and  storage 
allocations.  The  preliminary  design  information  to  be  reviewed  was  documented 
and  delivered  to  the  SPO  for  advance  examination,  prior  to  each  PDR.  Results 
of  the  reviews  were  subsequently  incorporated  in  PDR  Minutes,  which  were  pre¬ 
pared  by  the  contractor  and  approved  by  the  SPO. 

6 .  Critical  Design  Reviews  (CDR) 

The  basic  approach  adopted  in  BUIC  III  was  to  conduct  CDR  incrementally  in 
association  with  PQTs.*  For  this  reason,  the  chart  in  Figure  5-1  shows  a 
single  row  of  combined  PQT/CDR  events  for  each  CPCEI.  The  precise  dates  and 
numbers  are  discussed  further  under  PQTs,  in  the  next  subsection. 

However,  in  the  case  of  SEP,  some  elements  underwent  CDRs  which  were  not 
associated  with  PQTs.  These  were  cases  in  which  combined  PQT/CDRs  had  been 
planned  initially,  but  for  which  PQT  requirements  were  subsequently  waived. 


*  ESD  Exhibit  EST-1  (Sec.  H;  Exh.  XIV)  defines  CDR  for  CPCEIs  as  basically  a 
review  at  the  level  of  logical  design  of  individual  CPCs,  prior  to  coding 
and  testing.  However,  it  also  permits  nflexible  application"  in  the  case 
of  CPCEIs  whose  development  follows  an  incremental  pattern  during  Acquisition, 
suggesting  that  the  CDR  may  be  held  in  corresponding  increments  and  may  be 
combined  with  PQTs. 
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The  SEP  portions  affected,  and  the  dates  of  their  CDRs,  are  listed  below: 


BEPS 


Exercise  Tape  Generator 

Exercise  Tape  Description  15  January  1968 

Exercise  Tape  Modification 


BARS 


Test  Data  Analysis  Processor  1  February  1967 


Where  PQTs  were  held,  material  reviewed  at  CDR  was  the  available  Part  II 
specification  material  for  the  elements  undergoing  PQT.  Hence,  it  was  at 
the  level,  in  essence,  of  completed  design.  It  is  reported  that  items 
specifically  reviewed  included,  for  each  given  CPC  or  combination  of  CPCs , 
the  following:  CPC  name(s),  functions,  size,  and  interfaces;  emphasis  was 
placed  on  CPC  sizes,  noting  comparisons  with  BUIC  II  and  deviations  from 
PDR  estimates.  It  has  also  been  reported  that  questions  of  timing  were 
raised  with  increasing  interest  as  the  program  progressed  towards  later  phases. 

7 .  Preliminary  Qualification  Tests  (PQTs) 

As  indicated  in  Figure  5-1,  PQTs  occurred  in  large  numbers  for  the  ADP  and  SEP 
CPCEIs.  Initial  planning  of  PQTs  was  based  on  the  philosophy  that  preliminary 
qualification  testing  was  required  for  each  and  every  element  of  the  computer 
program.  Accordingly,  they  were  so  defined  and  scheduled  in  the  ADP  and  SEP 
Category  I  Test  Plans. 

As  was  noted  in  the  preceding  subsection,  each  PQT  was  initially  planned  to 
occur  in  combination  with  an  incremental  CDR  of  the  element (s)  being  PQT'd. 
Detailed  data  relating  to  these  events  as  they  actually  occurred  are  set  forth 
below,  separately  for  each  of  the  CPCEIs. 


AIR  DEFENSE  PROGRAM  (ADP) 


The  record  shows  that  a  total  of  35  PQTs  were  held  for  the  ADP,  of  which  32 
were  combined  with  CDRs.  Listed  below  are  the  dates  of  occurrence,  and 
following  each  the  number  of  PQT/CDRs  held  on  that  date.  The  last  three  dates 
listed  are  the  occasions  on  which  PQTs  were  not  accompanied  by  CDRs. 


19  September  1966  (2)  26  July  1967  (2) 


13  September  1967  (2) 


1  November  1966 


22  August  1967 


14  September  1967  (3) 


7  March  1967  (4) 


23  August  1967 


10  October  1967 


25  July  1967 


24  August  1967 


11  October  1967  (3) 


25-26  July  1967 


12  September  1967  21  November  1967  (3) 
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5  December  1967  (3)  16  January  1968  26  February  1968 

15  December  1967  (2)  30  January  1968 

In  some  cases,  a  single  PQT/CDR  was  held  for  an  entire  CPC  (see  Table  5-5). 
For  selected  components  of  ADP,  CDRs  were  conducted  in  increments  as  the 
design  and  testing  of  pieces  of  a  CPC  progressed.  In  the  case  of  incremental 
CDRs,  areas  of  a  computer  program  which  did  not  have  a  PQT  at  the  first  CDR 
for  that  computer  program  had  a  PQT  at  a  subsequent  CDR.  Thus  some  CPCs 
underwent  as  many  as  three  CDRs.  Table  5-5  shows  the  number  of  CDRs  held 
for  each  CPC  in  the  ADP  CEI  and  in  the  order  they  occurred.  The  Minutes  for 
each  CDR  list  the  subroutines  and  "functional  areas"  of  the  CPC  which,  at  a 
particular  time,  did  or  did  not  have  a  PQT  and  which  did  or  did  not  undergo  a 
CDR. 


Table  5-5.  PQTs/CDRs  for  ADP  CPCs 


No.  of 
CDRs 

No.  of 
CDRs 

No.  of 

CDRs 

1. 

MIN 

- 

2 

ii. 

TRK 

- 

i 

21. 

SIG  - 

1 

2. 

BIT 

- 

1 

12. 

RAC 

- 

3 

22. 

TAR  - 

1 

3. 

BOB 

- 

3 

13. 

MID 

- 

1 

23. 

SET  - 

0 

4. 

AAD 

- 

3 

14. 

KAW 

- 

1 

24. 

CIO  - 

* 

5. 

BIP 

- 

2 

15. 

RAP 

- 

2 

25. 

KAS  - 

* 

6. 

OCS 

- 

1 

16. 

COP 

- 

1 

26. 

SRT  - 

* 

7. 

REC 

- 

1 

17. 

SID 

- 

1 

27. 

ART  - 

* 

8. 

BID 

- 

1 

18. 

TAC 

- 

1 

28. 

SQR  - 

* 

9. 

TOP 

- 

1 

19. 

TAG 

- 

1 

29. 

TNC  - 

* 

10. 

BIO 

— 

2 

20. 

BOK 

— 

1 

30. 

WND  - 

* 

It  should  be  noted,  as  shown  in  Table  5-5,  that  one  CPC,  SET,  in  the  ADP  CPCEI 
did  not  have  a  CDR,  and  seven  CPCs  (CIO,  KAS,  SRT,  ATR,  SQR,  TNC,  and  WND) 


*  These  CPCs  underwent  CDRs  as  parts  of  other  CPCs. 
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Table  5^6.  Numbers  of  ADP  PQT  Procedures  and  Reports  by  Functions 
and  CPCs 


Function  or  CPCs 

Procedures 

Renorts 

i. 

Radar  Inputs 

2 

3 

2. 

Active  Tracking 

1 

3 

3. 

Passive  Tracking 

2 

4 

4. 

Height 

2 

3 

5. 

Positive  Target  Control 

3 

3 

6. 

Identification 

3 

5 

7. 

BOMARC  Guidance 

4 

4 

8. 

Manned  Interceptor  Guidance 

4 

4 

9. 

BOMARC  Prelaunch 

2 

2 

10. 

TDLL 

1 

1 

11. 

Air  Defense  Artillery 

2 

6 

12. 

Information  Transfer 

6 

6 

13. 

Manual  Inputs 

2 

2 

14. 

Startover 

1 

0 

15. 

Control 

1 

1 

16. 

Recording 

1 

1 

17. 

Simulation  Information  and  Tape  Read 

3 

3 

18. 

Tabulation  and  Situation  Displays 

4 

4 

19. 

Weapons  Assignment  and  Commitment 

4 

4 

20. 

Weapons  Illegal  Switch  Action 

1 

2 

21. 

Surveillance  Illegal  Switch  Action 

1 

2 

22. 

Illegal  Switch  Actions 

2 

0 

Totals  52 

63 

63 


underwent  a  CDR  and  testing  as  parts  of  the  CPCs  utilizing  them.  A  total  of 
32  CDRs  were  conducted.  Those  PQTs  conducted  in  January,  February,  and  March 
1968  for  CPCs  which  had  previously  undergone  CDRs  did  not  have  CDRs  a  second 
time. 

Although  participants  report  that  the  CDR  discussions  did  result  in  a  number 
of  actions,  including  design  changes,  essentially  no  actions  are  recorded  in 
the  CDR  minutes.  This  circumstance  is  apparently  a  function  of  the  fact  that 
minutes  were  prepared  only  once,  after  the  18-month  series  of  CDR  increments 
was  completed.  One  can  guess  that  the  record  of  actual  events  would  have  been 
better  had  the  minutes  also  been  prepared  in  corresponding  increments. 

For  the  ADP  CPCEI,  a  total  of  53  PQT  Procedures  documents  and  63  Test  Reports 
were  published.  The  General  PQT  Procedures  document  was  issued  on  23  June 
1966,  with  the  balance  issued  during  the  remainder  of  1966  and  throughout  1967 
and  1968.  The  earliest  Test  Reports  were  issued  in  October  and  November  1966, 
with  the  balance  of  the  reports  widely  distributed  throughout  1967  and  the 
months  of  January,  February,  and  March  1968. 

The  large  number  of  Procedures  documents  (52)  is  due  to  the  fact  that  for  some 
functions  several  PQT  procedures  were  prepared,  as  shown  in  Table  5-6.  The 
table  shows  the  number  of  Procedures  documents  issued  for  each  function 
identified  in  the  Part  I  specification.  It  should  be  noted  that  the  Pro¬ 
cedures  and  Test  Reports  do  not  match  the  functions  in  the  Part  I  specifi¬ 
cation  on  a  one-to-one  basis  in  many  cases,  but  are  also  related  to  CPCs. 

The  table  also  shows  the  number  of  Test  Reports  (63)  for  the  same  functions  or 
CPCs.  In  some  cases  there  are  more  Reports  than  Procedures  as  a  result  of 
test  re-runs. 


SYSTEM  EXERCISE  PROGRAM  (SEP) 


According  to  the  formal  CDR  Minutes  for  the  BARS  portion  of  SEP,  PQTs/CDRs 
were  conducted  on  the  dates  listed  below.  The  number  in  parentheses  shows 
the  number  of  CPCs  for  which  PQTs/CDRs  were  held  on  that  date. 

System  Test  Reduction  Processor 


11  December  1967  (3) 

12  December  1967  (4) 


16  January  1968  (3) 

17  January  1968 
13  March  1968 


13  December  1967 
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Operational  Processor 

11  December  1967  (5)  13  March  1968 

14  December  1967  14  March  1968 

18  January  1968  (2) 

Exercise  Processor 

14  March  1968 

Various  materials  were  made  available  for  use  as  necessary  in  accomplishing 

the  CDR  for  each  CPC.  These  included  the  Part  I  specification,  a  list  of  ADP 

items  and  tables  used,  and  drafts  of  the  Part  II  specification  including 
programs  listings,  and  users  manuals.  The  list  of  items  and  tables  used  for 
each  computer  program  or  functional  area  was  usually  part  of  the  PQT  Pro¬ 
cedures  document. 

For  the  SEP  CPCEI ,  a  total  of  30  Procedures  documents  and  27  Test  Reports  were 
published.  The  General  PQT  Procedures  document  was  issued  on  23  November  1966. 
Others  were  issued  over  a  six-month  period  from  October  1967  to  March  1968. 

The  Test  Reports  were  published  in  June,  September,  and  December  1967,  and  in 
January  and  March  1968. 


UTILITY  COMPUTER  PROGRAM  (UCP) 

For  the  UCP  CPCEI,  Table  5-7  shows  the  dates  for  the  conduct  of  PQTs/CDRs,  and 
the  dates  of  issue  of  the  PQT  Procedures  and  Test  Reports.*  The  SPO  did  not 
send  representatives  to  the  CDRs  conducted  in  conjunction  with  the  PQTs  on 
1  August  and  30  July  1967. 

Table  5-7.  Dates  of  UCP  PQTs  and  Basic  Issues  of  Procedures  and  Reports 


Functions 

Procedures 

PQTs/CDRs 

Reports 

JOVIAL 

1  July  1967 

1  Aug.  1967 

1  Sept  1967 

Adaptation  Calculation  (BAC) 

30  June  1967 

30  July  1967 

30  Aug.  1967 

*  Out  of  a  total  of  67  UCP  CPCs,  only  8  underwent  a  PQT.  These  include  7  CPCs 
for  the  Compiler  and  1  CPC  for  BAC. 
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8. 


Formal  Qualification  Test  (FQTs) 


As  indicated  in  Figure  5-1,  FQTs  were  performed  only  for  the  ADP  and  SEP  con¬ 
tract  end  items.  In  both  cases,  FQTs  were  conducted  in  a  number  of  separate 
parts,  as  identified  in  the  list  below.  A  separate  FQT  Procedures  document 
and  Test  Report  were  written  for  each  part. 

AIR  DEFENSE  PROGRAM  (ADP) 

Simulation  Mode  Test 
Interface  Test 
Live  Mode  Test 
System  Load  Test 

SYSTEM  EXERCISE  PROGRAM  (SEP) 

BEPS:  Exercise  Preparation  System 

BARS:  Exercise  Processor 

Operational  Processor 

System  Test  Reduction  Processor 

Subsystem  Test  Processor 

The  initial  FQTs  for  all  parts  were  conducted  at  the  Category  II  site  (BUIC 
Evaluation  Facility  (BEF) ,  located  at  Hanscom  Field,  Massachusetts)  during  the 
period  2  April  through  16  April  1968,  with  the  exception  that  the  Subsystem 
Test  Processor  had  been  FQTfd  at  the  contractor’s  plant  on  12-14  September 
1967. 

Because  of  a  major  deficiency  relating  to  cycle  time  of  the  ADP,  which  had 
been  known  to  exist  at  an  earlier  date  but  had  not  been  overcome  by  the  time 
of  the  April  FQT,  a  complete  re-run  of  the  ADP  FQT  was  scheduled  and  performed 
in  December  1968  to  verify  corrections  which  had  been  accomplished  by  that 
time.  The  symbol  shown  for  that  month  for  ADP  in  Figure  5-1  represents  this 
re-run . 

Scheduled  additional  FQTs  at  future  dates  may  also  be  noted  in  the  chart. 

These  represent  (a)  a  major  new  version  of  ADP,  with  corresponding  changes 
reflected  in  SEP  components,  to  incorporate  new  operational  capabilities 
beyond  those  for  which  requirements  had  been  established  earlier  in  Acquisition 
and  (b)  a  scheduled  updating  of  CPCEIs  associated  with  turnover  of  BUIC  III 
to  the  user. 
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9. 


Part  II  Specification 


The  Part  II  specification  for  each  of  the  three  CPCEIs  was  subdivided  into 
an  extensive  set  of  volumes.  Each  had  a  first,  General  volume  describing 
the  design  and  structure  of  the  CPCEI  as  a  whole.  Additional  volumes  were 
devoted  to  individual  computer  program  components  (CPCs),  for  the  most  part, 
although  a  few  dealt  with  data  or  other  special  items  such  as  the  Compool 
Description,  Constants,  Adaptation,  and  Library. 

The  30  CPCs  of  the  ADP  are  listed  in  Table  5-8  as  an  example.  The  ADP 
specification  contained  38  volumes  in  all,  while  there  were  141  volumes  for 
SEP  and  66  for  UCP.  As  a  complete  set,  the  Part  II  specifications  for  the 
three  CPCEIs  contain  approximately  40,000  pages  of  computer  program  design 
documentation.  Drafts  of  some  volumes  of  the  Part  II  specifications  were 
issued  as  early  as  February  1967.  Many  of  the  drafts  were  revised  and 
reissued  as  many  as  3  or  4  times  prior  to  the  date  of  15  April  1968,  when  all 
volumes  for  the  three  CPCEIs  were  delivered  for  review  prior  to  FACI.  The 
15  April  date,  as  shown  in  Figure  5-1,  therefore  represents  the  first 
t'completion,,  date  for  the  Part  II  specifications.  However,  they  were  still 
pre-FACI  drafts  at  that  time,  rather  than  basic  issues. 

Potential  problems  of  Part  II  specification  timing  in  relation  to  FQT,  FACI, 
and  Category  II  testing  had  been  recognized  early  in  the  program.  In  reaching 
a  solution  to  these  problems,  account  was  also  taken  of  the  additional  factors 
introduced  by  the  on-going  processing  of  changes.  The  solution  adopted  is 
outlined  briefly  below. 

An  artificial  "freeze"  of  the  computer  program  was  instituted  to  reflect  com¬ 
pletion  of  the  Part  II  specification  drafts,  shortly  prior  to  their  issue 
date  of  15  April.  This  configuration  represented  Version  1  of  each  CPCEI, 
which  was  the  version  to  be  FACl’d.  Subsequently,  the  drafts  would  be 
re-published  as  basic  issues  of  the  Part  II  specification — constituting  the 
initial  baselines,  but  would  contain  no  updating  or  corrections  other  than 
those  resulting  from  SPO  comments  at  FACI.  As  time  passed  and  the  intervening 
events  occurred,  these  basic  issues  were  eventually  completed  (for  ADF)  in 
February  1969.  Except  for  a  few  volumes,  they  had  continued  to  reflect 
Version  1  in  its  frozen  configuration. 

However,  changes  continued  to  occur,  in  actuality,  throughout  that  period. 

ECPs  had  been  in  process  at  the  time  of  the  freeze,  and  more  changes  resulted 
from  FQT,  Category  II  testing,  and  other  causes.  One  major  set  of  changes 
was  introduced  to  overcome  the  cycle  time  problem  (see  next  chapter)  in  ADP; 
and  Version  7  of  ADP  underwent  a  second  FQT  to  test  those  changes  in  December 
1968.  All  changes  through  Version  7  were  formally  processed  and  issued, 
simultaneously  with  the  basic  issue,  as  product  configuration  baseline  changes 
in  February  1969.  Thus,  in  effect,  the  initial  product  configuration  baseline 
was  established  with  the  issuance  of  pre-FACI  drafts. 
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Beyond  the  date  of  pre-FACI  drafts,  additional  events  depicted  in  Figure  5-1 
represent:  (a)  completion  of  basic  issues  for  SEP  and  UCP  late  in  1968;  (b) 
the  major  updating  of  ADP  in  February  1969;  and  (c)  a  projected  complete 
revision  of  all  Part  II  specifications  at  the  time  of  turnover  to  the  user 
in  April  1970. 

10 .  First  Article  Configuration  Inspection  (FACI) 

The  first  FACI  events  shown  in  Figure  5-1  represent  the  FACI  which  was 
accomplished  on  20-24  May  1968,  approximately  a  month  following  delivery  of 
pre-FACI  drafts  of  the  Part  II  specifications,  for  all  three  CPCEIs.  One 
element  of  SEP,  the  Subsystem  Test  Processor,  which  had  previously  undergone 
an  in-plant  FQT,  had  also  passed  an  in-plant  FACI  in  September  1967.  FACI 
was  completed  at  that  time  for  all  other  elements  of  the  CPCEIs,  except  for 
5  volumes  of  the  ADP  specification  dealing  with  CPCs  which  were  undergoing 
known  major  changes  associated  with  the  cycle  time  problem.  As  a  result, 
these  were  reserved  for  a  subsequent  FACI  "increment"  which  was  held  following 
delivery  of  Version  7  (the  Timing  Version)  of  ADP  in  February  1969. 

11 .  Category  II  Testing 

The  Category  II  tests  for  BUIC  III  were  conducted  at  the  BUIC  Evaluation 
Facility  (BEF),  located  at  Hanscom  Field,  Massachusetts,  during  the  4-month 
period  of  15  April  to  15  August  1968.  The  BEF,  which  was  created  as  an  ESD/ 
MITRE  test  facility  at  Hanscom,  was  configured  as  an  operational  BUIC  III  site. 
Following  Category  II  tests,  it  was  closed  on.  16  August,  but  was  later 
re-opened  for  additional  testing  associated  with  BUIC  III.  This  additional 
testing  included  the  repetition,  in  December,  of  the  Timing  Version  FQT  for 
the  ADP,  except  for  the  Live  Mode  Test  which  was  conducted  at  operational 
site  Z-10,  North  Truro,  Massachusetts. 
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CHAPTER  VI 


DISCUSSION  AND  EVALUATION 


A.  GENERAL 

1.  Qualifying  Factors 

BUIC  III  has  been  referred  to  as  a  "test  bed,"  in  which  the  novel  elements 
described  in  Section  C  of  the  preceding  chapter  are  thought  of  as  the  experi¬ 
mental  variables  whose  effects  on  the  system  program  are  being  identified  and 
evaluated.  From  that  point  of  view,  it  is  one  purpose  of  this  report  to 
assess  the  degrees  of  success  and  failure  associated  with  the  BUIC  III  applica¬ 
tions  of  those  management  techniques,  and  to  evaluate  them  accordingly.  How¬ 
ever,  it  should  be  recognized  that  the  mere  presence  of  the  requirements  did 
not  automatically  insure  that  they  all  became  effective  influences  in  the 
program.  In  fact,  there  were  circumstances  and  constraints  operating  through¬ 
out  to  preclude  simple  interpretations  in  terms  of  success  or  failure  as 
judged  against  the  stated  requirements. 

The  very  novelty  of  the  requirements  meant  that  many  of  their  complex  implica¬ 
tions  for  planning  and  implementation  were  not  readily  understood  and  appreciated. 
Supplemental  guidance,  interpretations,  and  explanations  to  accompany  the 
formal  statements  of  the  requirements  were  minimal  or  nonexistent;  and  there 
was  a  corresponding  absence  of  experience  upon  which  to  base  realistic  planning. 
The  detailed  implementing  solutiorts  were  worked  out  as  the  program  progressed, 
with  the  net  result  that  some  of  the  requirements  were  actually  implemented  in 
much  the  way  that  might  be  expected;  some  tasks  were  relatively  unaffected; 
and  others  managed  to  reflect  varying  degrees  of  compromise  with  tradition,  or 
to  take  unique  and  unexpected  forms. 

The  Category  I  Test  Plans  were  outstanding  products  of  that  initial  inexperience 
and  confusion,  as  the  first  to  be  contended  with  among  the  many  novel  elements 
associated  with  Category  I  testing.  The  writers  of  the  plans  attempted  to 
define  the  spectrum  of  requirements  and  planning  factors  for  the  computer  pro¬ 
gram  CEIs  in  fine  detail.  The  Part  I  specification  Sections  4  were  written 
from  the  plans,  rather  than  the  reverse.  Functions  of  the  plan  were  not 
distinguished  from  those  of  PQT  and  FQT  Procedures  documents  to  follow; 
planning  centered  around  functional  elements  of  the  Part  I  specifications  was 
not  clearly  relatable  to  structural  components  of  the  computer  programs;  and 
the  planning  often  tried  to  encompass  the  entire  gamut  of  computer  program 
test  activities  without  regard  to  distinctions  between  internal  development  and 
formal  testing  aspects.  The  resulting  bulk,  detail,  and  misconceptions 
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incorporated  in  these  plans  (for  ADP  and  SEP)  set  the  stage  for  many  troubles 
which  ensued  in  carrying  out  the  Category  I  testing  efforts  (see  Section  E 
below) . 

The  fact  that  BUIC  III  was  a  modification  of  BUIC  II,  rather  than  a  new  system, 
was  a  further  factor  which  interfered  with  adopting  some  of  the  new  techniques 
in  their  intended  form.  It  was  a  general  requirement  in  the  program  to  utilize 
existing  BUIC  II  content  and  materials  as  much  as  possible.  Conceptual  phase 
studies  were  limited,  for  the  most  part,  to  the  new  features;  and  there  was 
no  formal  Definition  phase.  Additionally,  it  should  be  noted  that  the  areas  in 
which  the  new  techniques  were  introduced  —  namely,  configuration  management, 
data  management  and  testing  —  did  not  include  system  engineering  management 
techniques  drawn  from  AFSCM  375-5,  with  the  relatively  minor  exception  of  design 
reviews.  Yet,  within  the  general  framework  of  375-series  concepts,  system  eng¬ 
ineering  is  recognized  as  the  fundamental  process  upon  which  other  techniques 
depend  for  their  substantive  content  and  execution. 

At  the  time  this  report  is  written,  various  problems  associated  with  adapting 
the  375-5  principles  to  data  processing  elements  of  systems  still  remain  to  be 
explored  and  resolved.  In  BUIC  III,  the  system  engineering  accomplished  was 
necessarily  of  an  ad  hoc  and  traditional  nature  —  which  meant  that  it  was  a 
combination  of  systematic  analysis,  trial-and-error  explorations,  and  previous 
experience,  with  the  technical  emphasis  naturally  devoted  to  areas  which  had 
proved  to  be  important  or  troublesome  in  earlier  systems. 

The  problems  encountered  with  cycle  time  of  the  operational  computer  program 
(see  Section  F  below)  can  be  largely  attributed  to  the  combined  effects  of 
these  various  factors  —  i.e.,  limited  Conceptual  phase,  no  Definition  phase, 
dependence  on  BUIC  II  precedent,  and  associated  absence  of  system  engineering 
analyses  which  were  sufficiently  comprehensive  to  detect  a  potential  timing 
problem  at  early  points  in  the  system  life-cycle.  In  fact,  the  effects  of  these 
factors  were  noticeable  throughout  the  Part  I  specifications.  Organization, 
style,  and  content  of  the  BUIC  III  Part  I  specifications  were  basically 
patterned  after  their  !,ops  spec"  predecessors,  even  though  they  carried  the 
new  titles  and  formats  dictated  by  the  Form  9. 

2 .  Program  Overview 

Discussions  of  the  specifications,  testing,  configuration  control,  and  other 
individual  topics  are  to  be  provided  in  subsequent  sections  of  this  chapter. 
Before  discussing  those  topics  individually,  however,  attention  is  directed 
briefly  in  this  section  to  the  series  of  events  as  a  whole,  based  on  their 
sequencing  during  the  Acquisition  phase  as  depicted  in  Figure  5-1  of  the 
preceding  chapter. 

Certain  general  impressions  of  the  total  program  can  be  gained  by  examining 
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the  actual  BUIC  III  sequence  of  milestones  in  the  light  of  expectations  based 
on  375-series  systems  management  concepts.  The  chart  may  be  compared,  in 
particular,  with  the  model  "road  maps"  contained  in  ESD  TR  68-1  for  the  com¬ 
puter  program  system  segment,  as  they  were  adapted  from  such  sources  as 
AFSCM  375-4  and  375-5. 

The  timing  of  events  during  the  first  year  of  Acquisition  appears  to  fit  fairly 
well  the  pattern  one  might  expect  from  a  system  program  having  no  Definition 
phase.  That  is,  ignoring  a  few  delays  in  particular  parts  of  the  items,  (a) 
the  System  Specification  initiated  both  the  development  of  Part  I  specifi¬ 
cations  and  preliminary  design  of  the  CEIs,  (b)  the  Category  I  Test  Plans  were 
closely  related  in  time  to  the  Part  I  specifications,  (c)  the  PDRs  were  held 
at  the  completion  of  preliminary  designs,  and  (d)  the  System  Specification 
underwent  a  major  update  and  expansion  as  a  result  of  the  Part  I  specification 
development  efforts. 

One  would  not  normally  expect  to  see  PQTs  held  on  so  many  occasions  as  they 
were  for  both  the  ADP  and  SEP  CEIs.  In  BUIC  III,  the  time  and  effort  expended 
on  PQTs  were  not  only  greater  than  expected,  but  were  also  far  more  costly 
than  the  results  justified.  The  problems  generated  by  this  abnormal  (one  hopes) 
PQT  program  are  discussed  at  length  in  section  E  below. 

The  timing  of  FQTs ,  at  their  first  appearance  immediately  preceding  the 
initiation  of  Category  II  tests,  may  be  considered  normal  for  operational  com¬ 
puter  programs,  although  it  would  be  later  than  normal  for  equipment  items. 
However,  the  continued  appearance  of  FQTs  over  a  time  span  of  one  and  one-half 
years  after  the  completion  of  Category  II  testing  is  a  surprising  circumstance 
for  any  type  of  contract  end  item.  These  continuations  are  identified  as 
arising  from  two  types  of  causes:  (a)  the  necessity  to  re-run  FQT  as  a  result 
of  deficiencies  in  meeting  the  original  requirements;  and  (b)  the  need  to  verify 
follow-on  versions  of  the  original  CEIs,  containing  changes  to  meet  extensive 
new  requirements.  Again,  the  questions  associated  with  this  and  other  aspects 
of  the  Category  I  test  program  are  discussed  further  in  Section  E  below. 

The  timing  of  Part  II  specifications  and  FACI  occurred  largely  as  planned  in 
relation  to  the  initial  FQT  and  Category  II  test  events,  except  for  (a)  a 
change  package  containing  major  revisions  of  the  Part  II  specifications  for 
ADP  associated  with  the  deficiency  noted  above,  and  (b)  a  delayed  "increment"  ot 
FACI  for  ADP  associated  with  the  same  deficiency.  The  chart  also  depicts  a 
planned  future  revision  and  reissue  of  the  specifications,  occasioned  by  turn¬ 
over  of  the  system  to  the  using  command. 

At  the  most  general  level,  it  seems  worthy  of  some  note  that  the  events  shown 
were  actually  incorporated  in  the  program  and  accomplished.  While  they  may 
now  be  taken  for  granted  in  the  light  of  the  past  few  years*  experience,  it 
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may  be  remembered  that  not  one  of  the  subsystem-level  milestones  listed  in  the 
chart  had  appeared  in  information  processing  system  programs  prior  to  BUIC  III. 
Collectively,  they  reflect  a  significant  trend  towards  increased  visibility  of 
elements  in  the  computer  programming  process,  in  the  direction  of  permitting 
SPOs  to  assume  increasingly  active  roles  in  the  management  of  computer  program 
acquisitions . 


B.  SPECIFICATIONS 

1 .  Part  I  Specifications 


The  influence  of  some  of  the  factors  mentioned  in  the  preceding  section, 
affecting  the  degree  to  which  formal  requirements  were  actually  implemented  in 
BUIC  III,  was  most  noticeable  in  items  produced  at  early  points  in  the  program. 
Thus,  while  a  considerable  effort  was  devoted  at  a  later  time  to  developing 
Part  II  specifications  which  would  conform  to  the  full  intent  of  the  Form  9, 
the  Part  I  specifications  retained  the  flavor  and  much  of  the  content  of 
their  BUIC  II  predecessors,  the  operational  specifications. 

The  operational  specifications  had  relied  heavily  on  narrative  description  of 
system  organization  and  operations.  Much  of  their  content  was  informational, 
rather  than  being  written  in  the  mandatory  language  of  a  "design-to"  specifica¬ 
tion.  In  some  ways  they  were  as  much  reports  of  the  analysis/definition  pro¬ 
cess  as  they  were  specifications  in  a  legal  sense.  For  example,  they  contained 
"scenarios"  of  system  data  processing  operations  which  often  included  human 
operator  actions.  While  such  scenarios  are  useful  system  engineering  devices 
associated  with  the  process  of  analyzing  and  identifying  functional  detail, 
and  constitute  levels  of  description  which  are  readily  understood  by  a  user, 
they  do  not  tend  to  provide  unequivocal  definitions  of  the  computer  program 
CEI  requirements. 

In  effect,  the  BUIC  III  Part  I  specifications  were  transitional  documents. 

They  contained  new  sections/subparagraphs  conforming  to  the  Form  9  in  such 
areas  as  interfaces,  inputs  and  outputs,  data  base,  human  performance,  and 
quality  assurance,  although  not  to  the  extent  of  resolving  a  variety  of 
questions  regarding  the  ways  in  which  some  of  those  topics  should  be  treated. 
The  comments  below  indicate  types  of  problems  which  have  been  encountered  in 
the  course  of  developing  Part  I  specifications  for  other  systems,  e.g.,  SEEK 
DAWN,  as  well  as  in  BUIC  III. 

a.  The  level  of  information  to  be  specified  in  the  Part  I  is 
relatively  well  defined  by  the  Form  9  (ESD  236)  in  some 
areas,  e.g.,  input  and  output  messages,  but  is  subject  to 
a  wide  range  of  interpretation  in  others.  The  inclusion 
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in  any  Part  I  specifications  of  many  mathematical  formulas 
and  equations  has  been  questioned  specifically,  where  they 
might  be  regarded  as  being  design  solutions  rather  than 
performance  requirements  as  such.  The  BUIC  III  Part  I's 
contained  many  of  these,  most  of  them  being  solutions  which 
were  known  as  a  result  of  BUIC  II  design  or  because  BUIC  III 
design  was  being  carried  out  concurrently  with  the  writing 
of  the  Part  I  specification.  Many  were  necessarily  incor¬ 
porated  as  design  requirements,  since  they  had  been  previously 
spelled  out  at  that  level  in  the  System  Specification,  also 
because  of  known  BUIC  II  design.  Otherwise,  and  with 
reservations  noted  elsewhere,  the  level  of  detail  represented 
in  BUIC  III  is  considered  appropriate  and  necessary  for 
adequate  definition  of  CPCEI  performance  characteristics  — 
that  is,  for  the  Part  I  specifications  in  their  completed 
form.  Generally,  much  of  the  required  information  would  not 
be  available  for  completing  the  specifications  at  that  level 
by  the  end  of  a  Contract  Definition  Phase  effort. 

b.  Information  called  for  under  paragraph  3.1.3,  "Data  Base 
Requirements,"  in  the  Form  9  is  relatively  clear  as  regards 
level  of  detail,  but  the  types  of  data  to  .be  specified  have 
proved  to  be  continuing  matters  of  debate.  The  ADP  specifi¬ 
cation  in  BUIC  III  confined  its  coverage  to  adaptation  data, 
which  are  specifically  called  out  in  the  Form  9.  Many  other 
types  of  data  were  specified,  but  under  the  various  functional 
elements  rather  than  under  the  data  base  paragraph.  They  in¬ 
cluded  computational  constants,  weather  data,  radar  inputs, 
weapons  characteristics,  and  others,  some  representing  data 
resulting  at  intermediate  stages  of  processing.  Different 
solutions  have  been  proposed  in  various  subsequent  cases, 
ranging  from  the  limited  treatment  exemplified  by  BUIC  III 

to  the  extreme  position,  as  in  the  SEEK  DAWN/818,  that  the 
data  base  paragraph  should  encompass  all  items  in  the 
computer  which  contain  data  values  as  distinct  from  com¬ 
puter  instructions.  However,  the  merits  of  these  different 
solutions  seem  to  involve  a  variety  of  technical  and  manage¬ 
ment  considerations  which  need  to  be  further  explored.  The 
general  problem  is  apparently  more  complicated  in  the  real¬ 
time  air  defense  context  than  it  would  be  for  some  other 
types  of  systems,  e.g. ,  473L,  in  which  the  computer  programs 
function  principally  to  retrieve  information  from  a  massive 
data  base. 

c.  Questions  have  also  been  raised  as  to  whether  the  Part  T 
specification  should  be  structured  to  permit,  or  require. 
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the  computer  program  to  be  designed  with  components  (CPCs) 
having  a  one-to-one  correspondence  with  the  Part  I  func¬ 
tional  elements.  Currently,  one-to-one  correspondence  is 
not  a  requirement,  for  the  reason  that  it  would  likely  in 
many  cases  restrict  the  freedom  to  strive  for  maximum 
efficiencies  in  computer  program  design.  However,  the 
absence  of  correspondence  also  creates  some  problems.  In 
BUIC  III,  the  most  noticeable  difficulties  were  in  the 
testing  area.  In  cases  where  CPCs,  or  groups  of  CPCs, 
were  not  directly  identified  with  given  elements  of  the 
Part  I  specification,  complications  were  met  in  the  planning, 
technical  preparation,  and  conduct  of  performance  testing. 

In  writing  the  Part  I  specification  for  SEEK  DAWN/818,  a 
related  problem  was  encountered  in  specifying  inputs  and 
outputs  for  the  functional  elements,  as  required  under  3.1. 2.1. 
Inputs  and  outputs  of  the  CPCEI  as  a  whole  are  necessarily 
the  same  at  the  Part  I  and  II  levels,  but  the  internal  inputs 
and  outputs  are  another  matter.  The  task  of  defining  all 
of  those  is  a  formidable  one,  in  itself,  and  it  can  prove  to 
be  of  dubious  value  where  the  functional  elements  undergo 
rearrangements  in  the  course  of  computer  program  design. 

While  a  matrix  can  be  constructed  (as  it  normally  is)  to 
depict  the  allocation  of  functional  elements  to  CPCs,  a 
similar  matrix  relating  internal  inputs  and  outputs  as 
between  the  Part  I  and  Part  II  levels  becomes  far  more 
complex.  To  illustrate,  using  the  BUIC  III  ADP  numbers 
of  15  functional  elements  and  30  CPCs,  there  can  be  105* 
sets  of  input-output  relations  among  functional  elements 
to  keep  track  of.  To  the  degree  that  re-structuring 
occurs,  many  of  these  would  not  be  included  within  the 
435  sets  (i.e.,  maximum  possible)  of  inter-CPC  inputs 
and  outputs,  since  some  number  of  functional  element 
inputs/outputs  would  be  contained  inside  CPCs.  Hence, 
many  are  lost,  many  are  scrambled,  and  many  more  new  inputs 
and  outputs  appear  which  are  not  identified  as  such  at  the 
Part  I  level.  In  the  real  case,  considerable  complications 
are  added  by  such  factors  as  variability  of  inputs  and 
interactions  among  components.  The  net  result  is  to  pose 
problems  of  design  and  development  which  may  not  be 
clearly  soluble,  as  well  as  to  complicate  the  testing  of 
individual  functional  elements.  Testing  of  the  integrated 


*  That  is,  for  K  elements  in  general,  there  can  be  as  many  as  K  (K-l)/2  non- 
duplicated  sets  of  input/output  interface  relations. 
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CPCEI  should  not  be  affected,  however,  since  the  identities 
of  external  inputs  and  outputs  are  presumably  maintained  in¬ 
tact  . 

d.  The  proper  content  to  include  under  the  human  performance 
paragraph  3.1.4  has  also  been  a  subject  of  frequent  dis¬ 
cussion.  A  "model 11  treatment  of  this  topic  has  not  yet 
appeared  among  the  Part  I  specifications  for  BUIC  III 
and  the  Southeast  Asia  systems.  A  deliberate  attempt  which 
was  made  in  SEEK  DAWN/818  did  result  in  a  few  examples  of 
appropriate  material.  The  examples  were  necessarily  limited 
and  to  some  degree  artificial,  however,  as  a  result  of  having 
to  be  formulated  after  the  designs  of  both  computer  program 
and  equipment  operating  stations  were  completed.  It  has 
been  observed  that  the  proper  content  of  the  human  performance 
paragraph  in  a  computer  program  Part  I  specification  is  not 
directly  analogous  to  what  it  would  be  for  equipment  items. 

In  the  case  of  equipment,  it  consists  largely  of  requirements, 
e.g.,  as  set  forth  in  MIL-STD-803,  which  govern  subsequent 
engineering  design  with  respect  to  workspace  layouts,  noise 
and  illumination  factors,  and  display/control  elements.  For 
computer  programs,  the  relevant  display,  control,  and  func¬ 
tional  requirements  associated  with  efficient  man-machine  de¬ 
sign  must  be  incorporated  into  the  detail  contained  within 
the  Part  I  specification  itself;  the  computer  programmer  has 
no  freedom  later  to  affect  those  solutions.  Thus,  if  the 
necessary  human  engineering  design  is  to  be  accomplished 
at  all,  it  must  be  done  as  an  integral  part  of  the  system 
engineering  effort  leading  to  the  Part  I  specification.  As 
yet,  in  the  systems  referred  to,  there  have  been  no  structured 
Conceptual  and  Definition  phases  which  might  have  provided 
opportunities  for  that  kind  of  systematic  development; 
and  much  of  the  process,  which  one  might  expect  to  be 
analogous  to  the  AFSCM  375-5  model  except  for  significant 
modifications  appropriate  to  the  data  processing  context, 
remains  to  be  explored  in  a  realistic  setting.  The  new 
system  programs  which  do  have  Definition  phases  should 
provide  additional  experience  in  this  area. 

2 .  Part  II  Specifications 

Requisite  information  to  be  contained  in  the  Part  II  specifications  for  the 

BUIC  III  CPCEIs  was  identified  by  the  Form  9,  ESD  237,  together  with  back-up 

sheets.  SDC,  early  in  the  development  effort  of  the  BUIC  III  computer  programs 


75 


prepared  and  submitted  to  the  SPO  an  example  Part  II  volume  portraying  the  level 
of  information  that  was  planned.  The  example  served  as  the  basis  for  arriving 
at  mutual  agreement  with  regard  to  the  volume  structure  and  level  of  informa- 
tion  content  of  the  Part  IIs.  The  agreements,  however,  did  not  prevent 
difficulties  which  subsequently  arose.  The  following  discussion  treats  the 
significant  aspects  of  the  Part  IIs  based  on  SDC's  experience.  The  discussion 
concentrates  on  the  Part  II  volumes  for  ADP  since  differences  peculiar  to  the 
Part  IIs  for  SEP  and  UCP  are  not  regarded  as  being  of  sufficient  significance 
to  warrant  explicit  treatment. 

The  Form  9  permits  structuring  the  Part  II  specifications  into  multiple  volumes. 
This  was  done  for  BUIC  III  and  done  quite  satisfactorily.  Volumes  1  through  8 
contain  information  essential  to  programming  tasks  performed  following  initial 
program  development  (e.g.,  effecting  program  changes,  adapting  the  computer 
program  to  unique  environments).  This  phase  is  normally  referred  to  as  the 
program  maintenance  phase.  No  unusual  difficulty  has  been  noted  in  reference 
to  these  volumes. 

By  contrast,  the  content  of  the  computer  program  component-oriented  Part  II 
volumes  has  provoked  some  discussion  and  concern.  Most  significant  are  the 
following: 

a.  Description 

b.  Flow  Charts 

c.  Interface 

d.  Limitations 

e.  CPC  Data  Organization 

In  this  context,  it  should  be  noted  that  the  format  and  content  of  these  CPC- 
oriented  Part  II  volumes  are  intended  to  satisfy  the  requirements  established 
by  paragraph  3.2.1  of  the  Form  9.  The  requirements  demand  a  brief  abstract 
of  the  functions  of  the  CPC,  the  languages  in  which  it  is  written,  and  its 
major  interfaces;  additionally,  the  requirement  states  that  the  CPC  shall  be 
described  in  detail  in  subparagraphs.  It  is  this  last  requirement  that  bears 
discussion  with  regard  to  Chapter  1,  Description,  and  Chapter  2,  Flow  Chart, 
of  the  CPC  Part  IIs  which  correspond  to  subparagraphs  3. 2. 1.1  and  3. 2. 1.2  of 
the  Form  9. 

An  examination  of  Part  II  volumes  reveals  that  the  presentation  of  descriptive 
detail  contained  in  Chapter  1,  Description,  varies  from  one  volume  to  another. 
This  is  to  be  expected  considering  the  varying  complexity  of  the  CPCs  and  the 
varying  ability  of  the  programmers  to  describe  the  CPCs  in  narrative  prose. 
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However,  the  point  to  be  made  here  is  simply  that  the  value  of  much  of  the 
detail  was  sufficiently  diminished  by  poor  writing  that  the  presumed  purpose 
of  the  description  was  not  satisfied.  That  is,  that  the  primary  subsequent 
function  of  the  description  is  to  convey  an  understanding  of  the  operations 
performed  by  the  subject  CPC  to  a  programmer  who  has  responsibility  for 
effecting  changes  and  corrections  to  the  CPC  in  the  maintenance  phase. 

A  similar  comment  applies  to  the  level  of  detail  presented  in  Chapter  2,  Flow 
Chart.  It  is  suggested  that  this  information  should  also  be  at  a  level  of 
detail  describing  the  operations  performed  by  the  CPC,  the  sequence  in  which 
they  are  performed,  and  graphically  portray  the  CPC  in  sufficient  detail  to 
enable  a  programmer  to  ascertain  which  region  of  the  CPC  performs  which 
operationa.  Thus,  the  flow  chart  should  provide  the  programmer  with  an  entree 
into  the  program  as  given  in  the  CPC  listing. 

Although  substantial  opinion  favors  flow  charts  with  gross  detail,  or  no  flow 
charts  at  all,  carefully  conceived  requirements  may  demand  detailed  flow  charts. 
If  such  is  the  case,  alternative  means  of  obtaining  or  satisfying  the  require¬ 
ment  should  be  examined.  The  possibility  of  obtaining  the  flow  charts  by 
means  of  an  automated  flow  chart  system  (i.e.,  a  special  purpose  computer 
program)  is  certainly  worth  considering.  Another  possibility  could  be  the 
substitution  of  a  form  of  logic  flow  table  instead  of  flow  charts.  The 
significant  advantages  of  either  of  these  approaches  must  lie  in  the  ease  and 
resultant  economy  with  which  they  may  be  employed  both  in  the  initial  pre¬ 
paration  of  the  flow  charts  (or  substitute)  and  their  subsequent  modification 
or  updating. 

An  automatic  flow  charting  capability  was  considered  in  the  BUIC  III  develop¬ 
ment.  However,  because  of  development  costs  and  other  factors  including  the 
detail  of  flow  charts,  the  proposed  capability  was  not  deemed  to  have  signi¬ 
ficant  advantages  over  manual  preparation  and  it  was  therefore  rejected.  This 
does  not  invalidate  the  concept  as  a  potential  approach  for  other  computer 
program  systems;  it  remains  a  possibility  worth  considering.  In  essence,  each 
CPC  is  described  three  times  within  the  Part  II  CPC  volume:  1)  prose;  2)  flow 
chart;  3)  listing. 

The  reason  for  concern  with  regard  to  the  level  of  detail  presented  in  the 
CPC  prose  description  and  flow  chart  is  not  only  the  effort  necessary  to 
initially  prepare  the  prose  and  flows,  but  the  effort  that  must  be  expended 
to  keep  them  current.  A  typical  schedule  for  initial  Part  IIs  might  be  as 
follows : 

1)  60  days  devoted  to  preparation  of  prose  and  flow  charts 
by  programmers 

2)  30  days  for  final  typing,  publication  and  delivery  to  SPO 
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3)  30  days  for  SPO  examination  preparatory  to  FACI. 

Program  development  consisting  of  coding  and  testing  is  normally  underway  con¬ 
current  with  the  preparation  of  the  Part  IIs.  Moreover,  both  tasks  are  per¬ 
formed  by  the  same  programmers.  Consequently,  the  level  of  detail  presented 
in  the  prose  and  flows  can  be  of  considerable  significance  since  programmers 
often  do  not  firm  the  fine  details  of  program  design  until  coding  and  initial 
testing  has  been  done.  If  the  fine  detail  must  be  firmed  120  days  prior  to 
FACI,  it  may  impose  undue  constraints  on  the  program  development  process  con¬ 
tinuing  to  FACI.  It  is  recognized  that  this  argument  is  not  always  true; 
difficulties  can  be  minimized  if  sufficient  allowance  is  made  in  the  schedule 
for  this  task,  but  fine  detail  can  be  costly.  Furthermore,  experience  has 
shown  that  command  and  control  systems  typically  undergo  considerable  change 
after  initial  development.  The  changes  may  be  necessitated  to  correct  pro¬ 
gram  errors,  remedy  design  deficiencies,  or  to  implement  new  requirements. 
Consequently,  the  greater  the  detail  in  the  prose  and  flows,  the  more  likely 
they  will  need  to  be  changed  to  maintain  identity  with  the  computer  program  as 
the  computer  program  undergoes  change.  Obviously,  the  cost  must  be  evaluated 
against  the  benefits  to  be  derived. 

Accordingly,  the  prose  can  describe  in  normal  English  such  information  as  the 
operational  tasks  of  the  CPC,  timing  considerations,  spares  considerations, 
and  unique  or  unusual  aspects  of  the  CPC  design;  the  flow  chart  can  identify 
the  region  of  the  CPC  that  performs  each  operational  task;  and  lastly,  the 
listing  represents  the  actual  configuration  of  the  CPC  accurately,  identically, 
and  completely.  The  prose  and  the  flow  chart  then  complement  one  another  and 
provide  a  means  of  informing  a  programmer  of  the  unique  algorithms  implemented 
by  the  computer  program  as  well  as  where  in  the  listing  he  should  look  for 
detailed  information.  Ultimately,  it  is  the  listing  to  which  a  programmer 
must  look  in  order  to  find  exact  information  of  how  the  CPC  performs  its 
operations  in  terms  of  actual  machine  instructions. 

The  chapter  on  interfaces  in  the  BUIC  III  Part  II  CPC-oriented  volumes,  cor¬ 
responds  to  subparagraph  3. 2. 1.3  of  the  Form  9.  These  chapters  did  not,  by 
themselves,  satisfy  the  interface  requirements.  Since  this  was  a  subject 
pursued  at  some  length  in  the  FACI  meeting  conducted  in  May  1968,  it  is 
evident  that  mutual  understanding  by  all  responsible  parties  had  not  been 
previously  attained.  The  question  was  essentially  whether  or  not  the  inter¬ 
face  requirements  had  to  be  treated  in  detail  within  each  CPC  volume  such 
that  each  volume  would  be  self-contained,  or  if  the  interface  requirements 
could  be  satisfied  by  reference  to  other  Part  II  volumes  containing  the  de¬ 
sired  information.  The  former  approach  would  have  included  all  interface 
information  relevant  to  each  CPC  in  the  individual  CPC  volumes.  This  would 
have  entailed  extensive  duplication  of  information  in  the  Part  IIs  as  a  whole 
which  would  not  have  particularly  enhanced  their  value.  Final  resolution  of 
this  question  was  obtained  by  agreement  that  references  would  suffice.  That 
is,  the  desired  information  was  contained  in  Part  II  volumes  such  as  Volume  4, 
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Compool  and  Layout,  Volume  3,  Compool  Description,  and  Volume  7,  System  Item 
Set/Use  Matrix;  additionally,  other  CPC  volumes  describing  subroutines  are 
referenced  as  necessary. 

The  approach  taken  in  resolution  of  the  question  appears  to  be  very  satis¬ 
factory  and  reasonable.  The  required  interface  information  is  contained  in 
the  Part  II  specification  in  a  form  that  can  be  readily  understood  by  competent 
programmers.  It  had  been  suggested  that  inclusion  of  the  interface  detail  in 
the  CPC  volumes  would  have  benefitted  inexperienced  programmers  charged  with 
the  responsibility  of  program  maintenance.  This  argument  is  not  necessarily 
valid;  in  the  case  of  BUIC  III  even  inexperienced  programmers  will  need  to 
know  how  to  use  Compool  Description  (Volume  3),  set/use  listings,  and  the  like 
in  order  to  fulfill  their  responsibilities.  Thus,  the  interface  requirements 
were  satisfied  by  the  special  purpose  volumes  in  conjunction  with  the  CPC 
volumes . 

Of  greatest  significance  with  respect  to  the  BUIC  III  Part  II  specifications 
is  the  question,  "How  much  of  what  was  prepared  is  necessary  and  useful?". 

It  was  the  consensus  of  contractor  personnel  that  the  net  result  of  following 
Form  9  requirements  to  the  letter  was  a  level  of  detail  below  that  which  would 
be  useful,  and  that  subsequent  specification  maintenance  at  that  level  would 
prove  to  be  impractical. 

This  experience  points  up  the  need  for  good  judgment  in  determing  a  proper 
level  of  detail  to  be  required  for  the  Part  II  specification.  In  general 
the  two  important  objectives  to  be  kept  in  mind  are  (a)  its  function  as 
an  instrument  for  configuration  control  and  (b)  its  use  by  computer  program¬ 
mers  as  an  aid  in  diagnosing  malfunctions  and  designing  modifications. 

Since  configuration  control  needs  are  met  most  directly  by  the  actual  listings, 
the  prose  and  flow  chart  requirements  should  be  directed  towards  attaining 
that  level  of  description  which  will  be  adequate  to  convey  an  understanding 
of  the  design  rationale,  nature  of  design  solutions,  and  the  flow  of  information. 

The  appropriate  level  of  detail  can  be  expected  to  vary  for  different  computer 
programs  and  systems,  as  a  function  of  such  factors  as  computer  program  nature 
and  complexity,  expected  frequency  and  magnitude  of  later  design  changes,  and 
provisions  for  operational  phase  computer  programming  support  and  specifica¬ 
tion  maintenance.  In  the  usual  case,  it  is  believed  that  the  minimum  detail 
required  to  accomplish  the  purposes  outlined  above  should  also  be  viewed  as  a 
desirable  maximum,  since  the  expense  of  not  only  producing  the  specification 
initially,  but  of  maintaining  it  later,  increases  significantly  as  additional 
detail  is  incorporated,  particularly  in  the  flow  diagrams. 
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C.  SPECIFICATION  CHANGES 

It  was  pointed  out  in  the  preceding  chapter  that  procedures  for  controlling 
changes  to  the  specifications  had  been  developed  and  applied  during  the  earlier 
SAGE  and  BUIC  II  programs.  As  regards  visibility  by  the  SPO  Configuration 
Control  Board  (CCB) ,  the  earlier  procedures  had  not  involved  formal  processing 
of  changes  at  the  Part  II  level,  but  had  been  confined  to  the  Part  I  (former 
"ops  spec")  level.  However,  experience  with  the  earlier  procedures  had 
created  a  sufficiently  firm  base  that  the  BUIC  III  configuration  control  and 
specification  maintenance  processes  were  handled  efficiently,  with  relatively 
minor  difficulties . 

Changes  were  initiated  by  issuance  of  a  Preliminary  Engineering  Change  Proposal 
(PECP)  to  the  CPCEI  specif ication (s)  involved.  In  cases  where  the  System 
Specification  was  affected,  a  PECP  was  accompanied  by  a  Specification  Change 
Notice  (SCN)  to  the  System  Specification,  but  not  by  SCNs  containing  proposed 
word  changes  to  the  CPCEI  specif ication(s) .  Instead,  the  PECP  described  the 
change  in  sufficient  detail  to  provide  a  basis  for  SPO  review,  approval,  and 
contractual  authorization.  Following  approval,  the  precise  word  and/or  data 
changes  were  then  developed  and  incorporated  into  specification  change  pages 
which  were  subsequently  issued  with  covering  SCN  and  formal  ECP.*  The  volume 
of  change  pages  was  such  that  they  were  normally  issued  periodically  in  pack¬ 
ages,  each  package  reflecting  changes  resulting  from  a  number  of  ECPs  and  CRs 
(Change  Reports,  for  Class  II  changes). 

Following  the  history  of  their  predecessors  in  SAGE  and  BUIC  II,  the  BUIC  III 
computer  programs  were  controlled  primarily  in  terms  of  changes  which  were  pro¬ 
cessed  at  the  design  requirements  level,  both  before  and  after  establishing 
product  configuration  baselines.  Class  I  changes  to  the  Part  II  specification 
only  were  limited  to  a  few  instances  in  which  formal  changes  were  processed  to 
cover  recompiling  of  the  computer  program  instructions. 

BUIC  III  also  followed  well-established  precedent  in  being  characterized  by 
frequent  and  extensive  changes  to  the  computer  programs  throughout  Acquisition. 


*  The  process  described  here  represents  a  basic  aspect  of  configuration  con¬ 
trol  for  computer  programs  as  embodied  in  ESD  Exhibit  EST-1,  which  departs 
significantly  from  the  accepted  change  processing  practice  for  items  of 
equipment.  Contractual  authorization  is  necessary  prior  to  the  issuance  of 
SCNs  for  the  reason  that  the  costs  associated  with  computer  program  changes 
are  totally  matters  of  R&D,  unaffected  by  such  factors  as  production  and 
logistic  support.  Costs  of  issuing  exact  changes  to  the  Part  I  specifica¬ 
tion  are  normally  appreciable;  once  the  product  baseline  has  been  established, 
the  total  cost  of  implementing  the  change  must  have  been  incurred  by  the 
time  all  specification  changes  are  completed. 
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Tables  6-1  and  6-2  below  list  the  numbers  of  Class  I  and  Class  II  changes, 
respectively,  which  occurred  in  the  Part  I  specifications  during  the  period 
March  1966  through  December  1968.  Changes  affecting  the  Part  II  specifica¬ 
tions  only,  which  occurred  after  product  configuration  baselines  were  est¬ 
ablished  (effectively,  April  1968),  are  summarized  in  Table  6-3. 

Change  page  packages  to  the  Part  I  specifications  were  issued  at  average 
intervals  of  about  6  weeks  between  May  1966  and  November  1968.  Figures 
indicating  the  volume  of  changes  pages  for  two  of  the  CPCEIs,  ADP  and  SEP, 
during  that  period  are  listed  in  Table  6-4.  For  comparison,  the  table  also 
lists  the  initial  and  ending  (November  1968)  numbers  of  pages  contained  in 
the  specifications.  Data  relating  to  how  much  of  the  material  on  the  pages 
was  changed,  or  on  what  per  cent  of  the  pages  were  actually  affected,  are 
not  available.  However,  it  may  be  noted  that  the  new  raw  numbers  are  large 
enough  that  each  of  the  original  specification  pages  might  have  been  replaced 
more  than  twice,  during  Acquisition. 

The  reasons  for  changes  were  naturally  many  and  varied.  Also,  the  requirements 
for  changes  originated  from  a  number  of  sources  other  than  the  contractor. 

While  the  information  is  not  complete,  sources  are  often  identified  in  the 
MECP  Detailed  Status  Summary  Forms”  which  were  contained  in  periodic  configura¬ 
tion  management  reports,  giving  descriptions  and  detailed  status  summaries  for 
each  ECP.  These  included  NORAD,  ADC,  MITRE,  Burroughs,  and  the  SPO.  In  all, 
nearly  50%  were  positively  identified  as  originating  from  such  sources,  largely 
based  on  needs  arising  from  changes  in  interfacing  equipment  and  systems. 


Table  6-1. 

ECPs  to 

the  Part  I 

Specifications 

Computer 

Program 

1966 

1967 

1968 

ADP 

46 

6 

17 

SEP 

26 

2 

1 

UCP 

15 

2 

1 

Totals 

87 

10 

19  =  116 
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Table  6-2.  Change  Reports  (CRs)  for  the  Part  I  Specifications 


Computer 

Program 

1966 

1967 

1968 

ADP 

37 

42 

42 

SEP 

24 

63 

15 

UCP 

5 

9 

2 

Totals 

66 

114 

59  =  239 

Table  6-3.  Part  II  Specification  Change  -  1968 


Computer 

Program 

ECPs 

CRs  i 

ADP 

i 

305 

SEP 

2 

167 

UCP 

1 

46 

Totals 

4 

518 

Table  6-4.  ADP  and  SEP  Part  I  Specifications: 
Number  of  Pages  vs.  Change  Pages 


Computer 

Program 

1966 

1968 

Change  Pages 

ADP 

2192 

2046 

5762 

SEP 

898 

866 

2682 
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D. 


DESIGN  REVIEWS  AND  FACI 


1 .  Preliminary  Design  Review 


The  Preliminary  Design  Reviews  for  the  three  BUIC  III  CPCEIs,  ADP,  SEP,  and 
UCP,  are  most  noteworthy  in  that  the  meetings  did  achieve  their  purpose;  a 
formal  technical  review  of  the  proposed  basic  design  approach  was  satisfacto¬ 
rily  accomplished.  For  each  CPCEI,  the  documented  preliminary  design  pre¬ 
sented  information  necessary  to  the  conduct  of  the  formal  review.  The  infor¬ 
mation  included  a  brief  review  of  the  interfaces  identified  in  the  Part  I 
Specifications,  a  discussion  of  timing,  a  concise  description  of  the  CPCs , 
storage  allocation,  and  other  information  pertinent  to  the  computer  program 
design.  This  was  essentially  in  accordance  with  system  management  guidelines 
contained  in  EST-1  and,  more  recently,  EST-3. 

The  ADP  PDR  was  significant  in  that  it  led  to  the  subsequent  cycle  time  analysis. 
The  expected  cycle  time  given  in  the  preliminary  design  exceeded  the  minimum 
specified  in  the  Part  I  Specification.  Since  the  input/output  table  size  was 
predicated  on  the  specified  minimum  time,  a  determination  was  made  that  the 
I/O  table  design  should  be  reconsidered  to  make  more  efficient  utilization  of 
the  output  capabilities.  The  ensuing  analysis  revealed  the  cycle  time  problem. 

Also  of  interest  is  the  fact  that,  contrary  to  the  understanding  obtained  at 
the  ADP  PDR,  JOVIAL  was  not  used  as  projected  for  the  development  of  ADP.  It 
was  intended  that  new  CPCs  or  rewritten  BUIC  II  CPCs  be  written  in  the  JOVIAL 
computer  program  language.  Subsequent  efforts  revealed  that  the  expansion 
factor  prohibited  the  use  of  JOVIAL;  code  written  in  JOVIAL  could  not  remain 
within  the  storage  constraints.  Consequently,  a  decision  was  made  to  forego 
JOVIAL  for  ADP  and  use  machine  language  instead. 

Thus,  it  is  evident  that  the  PDR  served  as  the  stage  for  involving  the  SPO/ 

MITRE  monitors  in  the  decision-making  process  with  respect  to  significant 
aspects  of  CPCEI  program  design.  The  design  agreed  to  at  the  PDRs  facilitated 
mutual  understanding  by  SDC  and  SPO/MITRE  of  the  problems  and  their  solutions 
in  the  computer  program  development  that  followed  the  PDR. 

2 .  Critical  Design  Review  (CDR) 

As  described  at  an  earlier  point  (Chapter  V) ,  CDRs  were  scheduled  and  held 
incrementally,  and  for  the  most  part  in  conjunction  with  preliminary  quali¬ 
fication  testing.  While  there  were  some  discrepancies  associated  with  the 
functional  emphasis  of  PQTs  as  opposed  to  the  computer  program  component  (CPC) 
emphasis  of  CDRs,  the  reviews  were  generally  concerned  with  design  documen¬ 
tation  for  the  elements  undergoing  test.  Hence,  the  reviews  were  held  at  the 
level  of  completed  design  for  those  elements. 
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Despite  their  close  association  with  PQTs,  however,  and  in  contrast  with  the 
PQT  experience  (see  Section  F  below) ,  there  is  a  notable  absence  of  reports 
of  difficulty,  resulting  changes,  or  other  action  items  related  to  the  conduct 
of  CDRs.  In  general,  the  CDRs  occasioned  very  little  concern  or  comment. 

On  the  whole,  these  BUIC  III  CDRs  did  not  appear  to  involve  elements  which 
were  particularly  novel  as  compared  with  prior  practice.  They  were  often 
attended  by  SPO  representatives,  and  typically  included  presentations  of  the 
design,  functions,  timing,  size,  and  interface  characteristics  of  the  CPC(s) 
being  reviewed.  However,  beyond  the  point  of  providing  visibility  of  technical 
progress,  the  purposes  apparently  did  not  extend  to  reaching  significant 
decisions  with  regard  to  the  development  cycle  of  the  computer  programs  involved. 
This  is  perhaps  not  an  unexpected  circumstance,  considering  that  the  designs 
were  already  complete  at  the  time  of  reviews. 

In  Exhibits  EST-1  and  EST-3,  the  emphasis  is  placed  on  CDRs  which  are  held 
’'at  the  level  of  flow  charts  or  computer  program  logical  design  prior  to 
coding  and  testing.”  At  that  level,  the  stated  formal  objective  is  to 
identify  the  design  documentation  which  will  be  released  for  coding  and 
testing.  Experience  on  computer  program  CDRs  in  that  category  is  not  avail¬ 
able  from  BUIC  III  or  other  system  programs  with  which  the  authors  have  been 
associated.  However,  computer  programmers  have  volunteered  firm  opinions  to 
the  effect  that  it  would  not  have  been  feasible  to  accomplish  the  stated  ob¬ 
jectives  of  a  CDR  at  that  level  for  a  CPCEI  as  complex  as  the  ADP.  Comments 
are  made  that:  the  design  which  initiates  coding  must  remain  flexible  during 
the  coding  process;  exact  interfaces  among  CPCs  are  not  visible  for  review  on 
an  incremental  basis;  and  the  ability  to  conduct  such  reviews  in  a  meaningful 
and  adequate  fashion  would  require  significantly  greater  technical  resources 
than  the  SPOs  have  typically  had  at  their  disposal. 

While  the  interim  design  level  (i.e.,  at  flow  charts)  is  given  the  primary 
emphasis,  it  is  to  be  noted  that  EST-1  does  provide  a  range  of  options  which 
is  broad  enough  to  cover  the  BUIC  III  application.*  Judging  from  the  comments 
which  have  been  made,  and  the  fact  that  they  did  not  contribute  additional 
problems,  it  may  be  fortunate  that  the  CDRs  were  handled  as  they  were,  in 
BUIC  III. 


3 .  First  Article  Configuration  Inspection  (FACI) 

The  first  Article  Configuration  Inspection  of  the  three  CPCEIs  was  essentially 
accomplished  in  May  1968.  The  FACI  was  noteworthy  in  that  there  was  very 
little  precedent  for  conducting  such  an  inspection  for  computer  program  items 
in  accordance  with  recognized  system  management  concepts.**  While  the  FACI 


*  ESD  Exhibit  EST-1,  Section  H,  p.  40-13. 

**  As  noted  elsewhere,  one  element  of  SEP  had  been  FACI’d  in  September  of  the 
preceding  year. 
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did  not  proceed  without  some  problems,  the  nature  of  the  problems  did  not  pose 
severe  obstacles  and  they  were  resolved  to  the  mutual  satisfaction  of  the  part¬ 
icipants.  The  objectives  of  the  FACI  were  achieved;  the  Part  II  specifications 
were  audited  and  approved  with  the  qualification  that  revisions  based  on  SPO 
comments  be  incorporated  in  the  basic  issues  of  the  Part  IIs  which  were  due  to 
be  published  by  1  February  1969.  With  SPO  approval,  the  Part  IIs  were  recognized 
as  the  instruments  defining  the  product  configuration  baseline  for  the  re¬ 
spective  CPCEIs.  For  practical  purposes,  however,  the  baseline  had  been 
established  in  the  preceding  month,  April,  at  the  onset  of  Formal  Qualification 
Tests.  Formal  control  was  maintained  of  all  changes  to  the  draft  Part  II  speci¬ 
fications  which  had  been  issued  describing  those  FQT  configurations. 

FACI  was  preceded  by  FQT  and  followed  by  Category  II  testing.  This  relative 
timing  is  reasonable  and  conforms  to  established  concepts.  However,  certain 
deviations  did  occur  as  a  result  of  the  known  cycle  time  deficiency  in  ADP . 
Although  an  approach  to  correct  the  deficiency  had  been  agreed  upon  and  approved, 
its  implementation  had  not  been  completed  prior  to  FQT  and  FACI.  Since  the 
corrections  entailed  extensive  changes  and  recompilation  of  five  computer 
program  components,  SDC  and  SPO/MITRE  agreed  to  defer  the  FACI  audit  of  the  Part 
II  volumes  for  those  five  CPCs.  The  audit  of  those  five  CPC  volumes  was  then 
performed  in  February  1969  after  implementation  of  the  cycle  time  changes,  thus 
concluding  the  FACI.  An  unusual  aspect  of  those  five  volumes  is  that  the  basic 
issues  include  Specification  Change  Notices  for  ECPs  and  CRs  that  were  pro¬ 
cessed  against  previous  draft  Part  II  volumes  of  the  five  subject  CPCs. 

The  contrast  of  the  approach  taken  to  the  FACI  meetings  for  BUIC  III  and  SEEK 
DAWN  Interface  Computer  Program  (SDICP)  is  interesting.  In  both  cases,  draft 
copies  of  the  Part  II  specifications  were  delivered  to  the  SPO  in  advance  of 
the  FACI  meetings;  BUIC  III  drafts. were  delivered  approximately  30  days  before 
FACI;  SDICP  drafts  were  delivered  approximately  90  days  before  FACI.  Whereas 
the  detailed  review  of  the  BUIC  III  Part  IIs  was  performed  at  the  FACI  meeting, 
the  review  of  SDICP  Part  IIs  was  essentially  done  prior  to  the  meeting.  In 
the  latter  case,  SPO/MITRE  comments  were  sent  to  SDC  so  that  mutual  agreement 
had  been  substantially  obtained  before  the  FACI  meeting.  The  meeting  thus 
served  to  confirm  understandings  that  had  already  been  reached  and  to  resolve 
those  few  questions  that  had  not  been  completely  resolved  earlier. 

The  difference  in  approach  is  of  interest  since  the  one  week  FACI  meeting  for 
BUIC  III  (disregarding  the  second  meeting  for  the  five  deferred  CPCs)  permitted 
only  a  superficial  examination  of  the  Part  II  specifications.  It  is  highly 
likely  that  a  more  comprehensive  examination  of  the  BUIC  III  Part  IIs  would 
have  revealed  many  more  discrepancies  among  the  CPC  descriptions,  flow  charts, 
and  listings.  Depending  on  the  bulk  of  the  Part  II  specifications,  available 
SPO  technical  resources,  and  other  relevant  factors,  the  SDICP  approach  may  be 
the  more  desirable  one  in  conducting  FACI  on  other  CPCEIs. 

The  prospect  of  a  FACI,  and  resulting  formal  control  at  the  level  of  product 
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configuration,  was  viewed  in  advance  with  some  apprehension.  While  procedures 
for  computer  program  control  and  accounting  had  been  in  effect  for  systems  pre¬ 
ceding  BUIC  III,  they  were  developed  and  used  internally  by  the  contractor,  had 
been  largely  confined  to  control  of  computer  program  listings,  and  had  not  been 
carried  out  under  the  formal  label  of  configuration  management.  However,  with 
minor  exceptions,  anticipated  difficulties  did  not  materialize. 

There  were  two  areas  of  potential  difficulty  which  deserve  mention.  The  first 
related  to  detailed  flow  charts  for  CPCs.  which  proved  to  be  expensive  and 
time-consuming  to  maintain  in  the  draft  Part  II  specifications  during  a  period  of 
some  months  preceding  FACI.  The  possible  continuing  impact  of  this  problem 
was  avoided,  following  FACI,  by  a  SPO-approved  change  which  eliminated  the  CPC- 
level  flow  charts  from  the  specification. 

A  temporary  problem  arose  in  the  course  of  Category  II  testing,  relating  to  a 
proposed  requirement  for  a  new  Version  Description  Document  to  cover  each 
daily  change  made  in  connection  with  the  test  activities.  The  question  of  what 
constitutes  a  "new  version"  is  not  unequivocally  defined,  and  disagreements 
occurred  on  the  point  among  personnel  of  the  SPO,  contractor,  and  test  team. 

It  developed  that  the  real  issue  was  more  a  matter  of  test  philosophy  than  of 
control  procedures,  and  working  solutions  were  reached  following  a  week  or  two 
of  discussion. 

However,  the  BUIC  III  experience  as  a  whole  has  provided  clear  evidence  that 
the  configuration  control  and  accounting  procedures  which  had  been  adapted  for 
computer  programs  were  remarkably  efficient.  Some  data  relating  to  rates  of 
changes  were  presented  in  the  preceding  section  of  this  chapter.  Following 
FACI,  control  was  extended  to  cover  the  computer  programs  and  Part  II  specifi¬ 
cations,  and  was  maintained  routinely  under  complex  circumstances.  In  practice, 
situations  arose  in  which  as  many  as  three  versions  of  the  Air  Defense  Program 
were  in  existence  or  under  development  at  one  time,  each  differing  signifi¬ 
cantly  from  the  others  with  respect  to  incorporation  of  approved  changes  and 
scheduled  introduction  into  test  or  operational  use.  The  process  included, 
for  example,  accounting  for  Class  II  error  corrections  made  to  a  given  version 
which  were  either  applicable  to  succeeding  versions  or  not  applicable  because 
of  superseding  Class  I  changes,  and  also  involved  keeping  track  of  change 
relations  with  other  computer  programs,  support  documentation  as  well  as 
specifications,  and  items  of  system  equipment.  Under  these  conditions,  con¬ 
trol  was  maintained  at  all  times  and  with  relatively  minor  difficulties. 
Considering  the  frequency  and  volume  of  changes  implemented,  it  seems  clear 
that  the  BUIC  III  experience  has  also  demonstrated  that  post-FACI  control  need 
not  seriously  impair  the  flexibility  with  which  computer  programs  can  be 
altered  to  accomplish  desired  changes  in  system  functions. 
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E. 


CATEGORY  I  TEST  PROGRAM 


When  the  Category  I  test  requirements  were  introduced  at  the  outset  of  the 
BUIC  III  program,  they  were  generally  viewed  as  innovations  which  would  have 
the  primary  effect  of  merely  formalizing  various  aspects  of  the  "normal”  test 
process.  Computer  program  test  and  evaluation  (CPT&E) ,  although  introduced  for 
the  first  time  as  a  new  term,  was  readily  recognized  as  an  inherent  and 
familiar  part  of  computer  program  development.  It  was  not  immediately  obvious 
in  advance  that  the  addition  of  a  few  preliminary  and  formal  qualification 
test  sessions  would  significantly  affect  the  total  computer  program  acquisition 
job  to  be  accomplished.  Nevertheless,  Category  I  testing  proved  to  be  an 
expensive  and  troublesome  area  throughout  the  BUIC  III  effort,  in  essentially 
all  of  its  various  aspects. 

The  record  indicates,  as  reviewed  in  the  preceding  chapter,  that  the  Category 
I  test  milestones — in  the  form  of  PQTs  and  FQTs  and  their  various  associated 
documents — were  actually  accomplished  during  the  program.  This  achievement 
now  seems  a  remarkable  one,  at  face  value,  although  it  is  conceded  to  have  been 
made  possible  only  by  the  fact  that  the  system  schedule  slipped  appreciably, 
for  independent  reasons,  during  the  critical  periods.  Throughout,  difficulties 
and  complexities  were  faced  by  SPO  and  contractor  personnel  in  interpreting, 
coordinating,  and  implementing  the  many  requirements.  In  fact,  the  effect  of 
resulting  confusions  and  misinterpretations  among  personnel  involved  in  the 
program  was  so  pervasive  that  an  unbiased  and  factual  account  of  the  problem 
is  difficult  to  reconstruct  on  the  basis  of  available  records  and  verbal 
reports . 

Preliminary  qualification  testing  of  the  ADP  is  the  specific  area  in  which 
troubles  are  universally  reported  as  being  most  important  and  acute.  As  out¬ 
lined  earlier,  a  total  of  35  PQT  sessions  were  conducted  on  19  different  dates 
during  the  15-month  period  of  September  1966  to  February  1968.  Associated  with 
these  35  PQTs  were  53  PQT  Procedures  documents  and  63  Test  Reports.  Although 
the  reasons  for  these  discrepancies  are  not  all  accounted  for,  it  appears  that 
some  PQTs  had  multiple  objectives  for  which  separate  procedures  were  written, 
some  PQTs  failed  and  had  to  be  re-run,  some  of  the  reports  had  to  be  re-written, 
and  a  few  scheduled  PQTs  were  never  accomplished. 

The  objective  established  for  PQTs  in  the  Category  I  Test  Plan  was  to  verify 
approximately  2500  "reactions"  of  the  computer  program  in  conformance  with 
detailed  requirements  which  had  been  set  forth  in  the  Part  I  specification. 

PQTs  were  planned,  as  a  complete  set,  to  provide  comprehensive  coverage  of 
nearly  the  whole  of  the  Section  3  requirements  for  the  ADP  CEI.  As  later 
described  by  contractor  participants,  some  (i.e.,  most)  of  the  PQT  sessions 
were  planned,  rehearsed  in  advance,  and  then  conducted  as  "demonstrations,"  in 
effect,  for  the  benefit  of  visiting  SPO  personnel.  These  were  generally  success- 
full,  but  were  also  time-consuming  and  more  costly  in  manpower  than  had  been 
anticipated.  Others  were  prepared  but  not  rehearsed,  and  were  actually  run 
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in  conformance  with  the  written  procedures  for  the  first  time  during  the  for¬ 
mal  PQT  meeting.  These  were  more  in  the  nature  of  true  "tests11  but  they 
often  failed  and  had  to  be  re-run  at  subsequent  times.  The  program  as  a  whole 
is  reported  to  have  caused  cost  overruns  and  difficulties  in  meeting  schedules, 
as  well  as  significant  morale  problems  among  many  computer  programmers,  who 
came  to  regard  the  PQT  requirements  as  monstrous  and  artificial  barriers  to 
their  progress  towards  completing  the  ADP  development  task. 

The  35  PQTs  which  were  carried  out  did  manage,  in  one  way  or  another,  to  result 
nevertheless  in  formal  verification  of  all  but  450  of  the  original  2500  re¬ 
actions  prior  to  the  beginning  of  FQT  and  Category  II  testing.  Of  those,  all 
but  110  were  subsequently  verified  at  the  Category  II  site,  and  the  remaining 
few  were  eventually  waived. 

Essentially  the  same  problems  were  encountered  in  PQTs  for  the  system  exercising 
CPCEI  (SEP),  although  the  totals  of  events  and  documents  were  slightly  less 
numerous:  24  PQTs,  30  Procedures  documents,  and  27  Test  Reports.  In  the 
utility  area  (UCP) ,  qualification  testing  for  the  entire  CPCEI  was  confined  to 
one  "PQT"  for  each  of  two  (out  of  a  total  of  25)  of  the  elements:  except  for 
these  two  tests,  the  entire  CPCEI  was  qualified  in  relatively  painless  fashion 
through  its  use  in  supporting  the  Acquisition  phase  development  of  ADP  and  SEP. 

The  formal  qualification  tests  (FQTs)  for  ADP  and  SEP  were  held,  as  noted 
earlier,  in  a  number  of  separate  test  sessions  in  each  case.  For  SEP,  the 
FQT  was  "incremental,"  in  the  sense  that  the  total  consisted  of  separate  test 
sessions  for  each  of  the  major  elements  of  the  CPCEI,  each  having  a  separate 
associated  plan,  procedures,  and  report.  This  arrangement  was  made  possible  by 
the  fact  that  those  major  elements  of  the  CPCEI  represent  capabilities  which 
can  operate  independently.  In  the  case  of  ADP,  four  separate  FQT  sessions 
were  run  to  examine  separate  aspects  of  the  performance  of  the  integrated 
CPCEI . * 

The  Procedures  documents  for  FQTs  posed  additional  difficulties  which  had  not 
been  typical  of  PQTs.  Whereas  PQT  Procedures  had  been  delivered  only  for  SPO 
review,  normally  about  30  days  in  advance  of  a  scheduled  test,  the  FQT  Pro¬ 
cedures  were  required  a  minimum  of  90  days  in  advance  for  review,  approval, 
and  revision  based  on  SPO  comments.  The  records  indicate  that  initial  drafts 
of  some  of  the  FQT  Procedures  for  ADP  were  revised  and  reissued  as  often  as 
five  times  before  being  accepted,  because  of  various  problems  involving  con¬ 
formance  to  the  Form  9,  the  style  of  presentation,  and  technical  adequacy. 

This,  as  in  the  conduct  of  PQTs,  was  an  area  in  which  the  project  personnel 
encountered  an  unexpectedly  firm  insistence  by  the  SPO  on  rigorous  adherence 
to  the  requirements,  and  in  which  inevitable  difficulties  resulted  from  a 


*  For  both  ADP  and  SEP,  all  test  sessions  were  run  on  an  identical  configura¬ 
tion  of  the  CPCEI. 
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general  absence  of  adequate  guidance  and  experience.  However,  the  necessary 
experience  was  acquired  in  the  course  of  this  troublesome  process,  to  the 
degree  that  a  subsequent  FQT  (see  below)  was  documented  and  accomplished  with 
significantly  fewer  perturbations. 

The  FQT  for  the  first  version  of  the  ADP  was  accomplished  immediately  prior  to 
Category  II  testing,  at  the  Category  II  site.  This  timing  and  location  had 
been  anticipated  from  the  outset  of  the  program,  based  on  a  variety  of  con¬ 
siderations  which  have  been  recognized  as  being  generally  pertinent  to  the 
qualification  of  such  complex  operational  computer  programs  as  the  ADP.*  How¬ 
ever,  as  was  noted  in  the  preceding  chapter,  a  second  FQT  was  performed  (the 
Timing  Version)  approximately  four  months  following  the  completion  of  Category 
II  testing,  and  a  third  and  fourth  are  planned  for  additional  follow-on 
versions  which  are  under  development  and  scheduled  for  delivery  at  future 
dates.  The  initial  version  of  ADP  which  underwent  FQT  prior  to  Category  II 
testing  contained  a  cycle-time  deficiency.  Interim  corrections  were  devised 
for  Category  II  use,  and  the  deficiency  was  removed  fully  in  the  Timing  Version 
for  which  FQT  was  then  repeated.  Thus,  one  might  assume  that  qualification  of 
the  CPCEI  had  been  accomplished  by  that  point  in  time.  Yet,  future  FQTs  are 
being  planned  for  the  follow-on  versions.** 

While  the  continued  repetition  of  FQTs  is  apparently  not  regarded  with  any 
particular  concern  by  either  the  SPO  or  contractor,  it  does  suggest  that  there 
are  certain  questions  regarding  the  philosophy  of  qualification  as  applied  to 
computer  program  contract  end  items  which  remain  to  be  clarified.  Judging 
from  SAGE/BUIC  II  precedents,  the  ADP  can  be  expected  to  undergo  continuing 
changes  which  will  probably  occasion  periodic  new  versions  at  the  rate  of 
three  or  four  per  year,  each  containing  additional,  altered,  and/or  deleted 
capabilities  as  compared  with  the  version  being  replaced.  In  SAGE,  each  new 
version  is  subjected  to  formal  tests  which  are  comparable  in  scope  and  level 
to  the  "FQTs"  planned  for  follow-on  Acquisition  phase  versions  of  the  BUIC  III 
ADP.  These  SAGE  tests,  however,  which  are  actually  conducted  by  the  using 
command  prior  to  introducing  each  new  version  into  operational  use,  are  known 
as  "acceptance"  tests.  One  might  question  the  application  of  either  term, 
acceptance  or  qualification,  to  such  tests  —  considering  that  "qualification" 
is  normally  confined  to  an  initial  article,  while  "acceptance  testing"  is 
firmly  associated  with  individual  units  of  a  CEI  undergoing  quantity  production 


*  The  considerations  are  mentioned  and  discussed  more  fully  in  ESD  Exhibit 
EST-1,  ESD  Supplement  1  to  AFSCM  375-4,  and  ESD-TR-68-1. 


**  It  is  being  suggested  here  only  that  this  situation  poses  some  novel  ques¬ 
tions  relating  to  the  general  concepts  of  qualification  and  acceptance 
testing,  not  that  the  repetition  of  FQTs  was  in  any  sense  "wrong";  in  fact, 
it  was  clearly  an  effective  and  appropriate  procedure  for  the  circumstances. 
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This  appears  to  be  an  area  in  which  the  type  of  computer  program  represented 
by  ADP  may  call  for  some  new  testing  concepts  which  are  not  readily  derived 
from  established  practice  with  equipment  items. 

It  was  mentioned  above  that  the  objective  of  PQTs  was  to  verify  some  2500  re¬ 
actions  of  the  computer  program,  covering  the  spectrum  of  detailed  requirements 
set  forth  in  Section  3  of  the  Part  I  specification.  While  it  is  to  be 
assumed  that  the  computer  program  must  be  designed  to  meet  those  detailed 
requirements,  and  must  "work”  in  the  operational  system,  the  sheer  number  of 
reactions  alone  suggests  that  the  task  of  formal  verification  can  be  one  of 
considerable  magnitude.  Many  of  the  reactions  are  interdependent;  they  should 
occur  under  variable  combinations  of  input  and  operating  conditions,  and  many 
can  be  verified  only  through  intricate  analysis.  This  much  was  evident  in 
varying  degrees  to  both  SPO  and  contractor  people  who  were  involved  in  planning 
the  Category  I  testing  at  the  outset  of  the  BUIC  III  program.  It  was  also 
obvious  at  that  time  that  the  total  job  could  not  be  accomplished  during  a 
limited  period  of  formal  qualification  testing  at  the  Category  II  site.  Based 
on  the  ground  rule  that  the  comprehensive  formal  verification  had  to  be  done, 
however,  it  was  decided  to  place  the  burden  of  verifying  detailed  reactions 
onto  the  PQTs,  and  to  limit  FQT  objectives  to  the  testing  of  higher-level 
characteristics  of  the  integrated  CPCEI  operating  under  system  conditions. 

This  decision  was  reached  early  enough  to  influence  the  writing  of  the  Category 
I  Test  Plans  for  ADP  and  SEP.  Together  with  some  lack  of  understanding  of  the 
test  plan  functions  in  relation  to  Section  4  of  the  Part  I  specification  on 
the  one  hand,  and  to  the  subsequent  Procedures  documents  on  the  other,  it 
accounts  for  much  of  the  bulk  and  detail  in  the  Category  I  Test  Plans. 

Thus,  the  groundwork  was  laid  for  the  troublesome  experience  which  ensued  in 
trying  to  carry  out  the  total  of  57  PQTs  for  ADP  and  SEP.  It  became  recognized 
early  in  the  process  by  both  SPO  and  contractor,  although  for  different  reasons, 
that  the  attempt  to  accomplish  formal  qualification  by  means  of  PQTs  was 
unsound  —  to  the  contractor  because  the  magnitude  of  the  task  has  not  been 
appreciated  and  reflected  in  planning  manpower  and  schedules,  and  to  the  SPO 
because  of  an  emerging  realization  that  such  testing  accomplished  during 
intermediate  stages  of  evolving  those  complex  and  massive  CPCEIs  would  provide, 
at  best,  a  weak  basis  for  guaranteeing  the  presence  of  specified  reactions  in 
the  "final"  product.  However,  the  contractual  commitments  reflected  in  the 
test  plans  were  enforced,  except  for  the  few  reactions  which  were  eventually 
waived  after  Category  II  testing  and  two  FQTs  for  each  of  the  major  CPCEIs  had 
been  completed. 

The  test  process  was  also  obviously  affected  by  the  continuing  flow  of  Part  I 
specification  changes  throughout  the  period  during  which  PQTs  were  being  con¬ 
ducted,  as  well  as  by  coding  changes  in  PQT’d  elements  as  they  were  assembled 
and  made  to  work  with  other  elements  of  the  computer  program.  The  desirability 
of  baselining  PQT'd  elements  at  the  instruction  level  was  considered  briefly 
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at  an  early  point,  in  the  context  of  qualifying  via  PQTs ,  but  was  discarded  as 
being  an  impractical  requirement  to  implement.  The  record  does  not  show  how 
many  PQTs,  or  tests  of  individual  reactions,  were  vitiated  by  subsequent  Part 
I  specification  changes,  or  how  many  were  added  as  new  requirements,  although 
it  may  be  inferred  from  the  number  of  Category  I  Test  Plan  changes  listed  in 
the  Configuration  Index  —  as  well  as  from  the  voluminous  flow  of  Part  I 
specification  change  pages  resulting  from  ECPs  —  that  the  numbers  were  sub¬ 
stantial.  These  changes  were  known  and  clearly  recognized  during  the  program. 
Also,  in  some  manner  which  was  never  readily  visible  to  an  outside  observer, 
the  contractor  was  able  to  implement  the  changes  and  maintain  an  updated  test 
effort  which  was  at  least  adequate  at  a  practical  level;  internally,  similar 
change  rates  in  SAGE  and  BUIC  II  had  become  a  way  of  life.  However,  there  is 
evidence  now  that  an  overt  recognition  of  this  "fluid"  nature  of  the  computer 
programs  in  question  was  conspicuously  missing  at  the  time  the  PQTs  —  and  the 
FQTs  too,  for  that  matter  —  were  initially  being  planned  in  BUIC  III.  In  re¬ 
trospect,  it  appears  that  the  testing  approach  adopted  would  have  been  more 

fitting  for  CEIs  having  less  complex  performance  requirements,  which  had  more 
stable  definitions  at  the  outset  in  terms  of  Acquisition  phase  end  objectives, 
and  which  could  be  expected  to  have  relatively  longer  operational  lives  in 
their  FACI'd  configurations. 

One  may  assume,  as  most  of  the  BUIC  III  personnel  actually  did,  that  the  trials 
and  tribulations  of  Category  I  testing  were  caused  by  application  of  the  new 
375-series  requirements.  This  was  undoubtedly  true  in  a  sense,  although  a 
step-by-step  study  of  the  requirements  —  i.e.,  of  the  Forms  9  and  exhibits  — 

fails  to  reveal  any  significant  errors  or  unsound  provisions  which  can  be 

pointed  to  as  being  responsible  for  the  troubles.  In  fact,  it  seems  clear  that 
the  major  difficulties  were  a  direct  result,  not  of  their  application,  but  of 
their  misapplication.  PQTs,  for  example,  should  never  have  been  planned  in 
the  first  place  as  formal  tests  which  would  actually  qualify  the  CPCEIs  with 
respect  to  the  full  ranges  of  requirements  contained  in  their  Part  I  specifi¬ 
cations  . 

However,  the  BUIC  III  experience  makes  it  equally  clear  that  the  existing  re¬ 
quirements  are  inadequate,  in  and  of  themselves  at  least,  to  guide  a  success¬ 
ful  Category  I  testing  program  for  such  CPCEIs  as  the  ADP  and  SEP.  As 
mentioned  earlier,  requirements  were  formulated  and  communicated  in  the  legal 
language  of  Forms  9  and  contractual  exhibits,  with  a  minimum  of  explanatory 
material  or  other  guidance  to  their  specific  interpretation  as  applied  to  the 
CPCEIs  in  question.  By  and  large,  the  requirements  are  sufficiently  general 
in  wording  to  cover  many  other  types  of  computer  programs,  including  those 
which  would  be  far  less  complex  and  would  remain  relatively  stable  during  both 
Acquisition  and  Operational  phases  of  their  existence.  By  virtue  of  this  very 
generality,  however,  they  are  deficient  in  both  guidance  and  specific  coverage 
for  the  classes  of  computer  programs  represented  by  ADP  and  SEP.  Although  it 
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is  beyond  the  purpose  of  this  report  to  construct  a  detailed  analysis  and  pro¬ 
pose  specific  changes,  the  points  discussed  briefly  below  will  attempt  to 
identify  the  problem  areas  more  directly,  and  to  suggest  better  interpretations 
which  might  have  been  made  to  the  requirements  as  they  now  stand. 

1.  The  degree  to  which  formal  verification  of  performance  requirements 
must  be  accomplished  during  Category  I  testing  is  a  matter  which  poses  special 
problems  in  the  case  of  many  CPCEIs.  Existing  guidance  tends  to  imply  that  all 
requirements  of  Section  3  in  the  Part  I  specification  must  be  satisfied.  Yet 
it  can  be  shown  that  for  the  ADP,  for  example,  if  one  takes  into  account  the 
numbers  of  defined  characteristics  and  their  countless  potential  interactions 
during  operating  conditions,  complete  verification  would  be  virtually  impossible 
to  achieve,  even  if  no  changes  were  permitted  for  the  life  of  the  item.  Hence, 

a  contractor  cannot  reasonably  commit  himself  to  the  task  of  proving  complete 
verification.  At  the  same  time,  a  SPO  obviously  needs  assurance  that  the  com¬ 
puter  program  does  meet  its  requirements,  at  least  well  enought  to  permit 
successful  operation  of  the  system.  This  dilemma  was  not  clearly  recognized 
in  BUIC  III,  and  many  of  the  difficulties  can  be  traced  to  the  supposition 
that  complete  verification  could  actually  be  accomplished.  Pending  the 
development  of  better  guidelines  in  this  area,  the  problem  seems  to  indicate  a 
need  for  good  judgment  in  the  SPO-contractor  relationship  to  assure  that 
qualification  requirements  are  established  on  a  practical  and  cost-effective, 
rather  than  perfectionist,  basis,  taking  account  of  both  the  CPCEl’s  inherent 
complexity  and  its  anticipated  degree  of  stability  as  an  element  of  the 
operational  system. 

2.  The  meaning  and  purposes  of  preliminary  qualification  testing  were 
interpreted  in  varied  ways  by  personnel  associated  with  BUIC  III.  Some  of  the 
confusion  might  be  ascribed  to  the  term  itself,  which  implies  qualification  as 
a  purpose;  and  the  available  guidance  is  sufficiently  scant  that  it  has  to  be 
analyzed  carefully  to  derive  the  intent  that  PQTs  are  really  preliminary  tc) 
qualification.  Beyond  this  bare  interpretation,  official  documents  presently 
provide  only  meager  amplification  —  although  there  are  some  isolated  re¬ 
ferences  which  might  be  used  as  starting  points.  For  example,  the  following 
statement  is  contained  in  ESD  Exhibit  EST-1 ,  Section  H:* 

"Since  the  total  process  [i.e.,  of  computer  program  development] 
is  typically  lengthy  . . . ,  and  represents  the  major  expense  of 
computer  program  acquisition  for  the  system,  it  should  normally 
include  preliminary  qualification  tests/demonstrations  at 
appropriate  stages  for  formal  review  by  the  procuring  agency. 

Requirements  for  such  tests  are  established  in  Section  4  of  the 


*  Section  H  is  a  portion  of  EST-1  which  did  not  appear  in  the  BUIC  III  con¬ 
figuration  management  exhibit. 


92 


Part  I  specification  and  are  amplified  in  the  contractor’s 
Category  I  Test  Plan.  While  the  tests  are  preliminary  in 
nature  (they  do  not  imply  acceptance,  or  formal  qualifi¬ 
cation)  ,  they  can  serve  the  necessary  purposes  of  providing 
check  points  for  monitoring  the  contractors  progress  towards 
meeting  design  objectives  and  of  verifying  detailed  per¬ 
formance  characteristics  which,  because  of  sheer  numbers 
and  complexity,  may  not  be  feasible  to  prove  in  their 
entirety  during  a  limited  period  of  formal  qualification 
testing. ” 

Elsewhere,  an  ESD  technical  report*  presents  the  concept  that  PQTs  can  be  re¬ 
lated  to  successive  stages  of  assembly  of  CPCEI  components  during  the  develop¬ 
ment  process.  The  authors  illustrate  a  "building  block"  approach  to  CPCEI 
development,  in  which  each  new  CPC  is  envisaged  as  being  assembled  to  the 
existing  piece (s)  of  the  CPCEI  and  each  assembly  stage  is  preceded  by  a  CDR 
(prior  to  coding  the  CPC)  and  accompanied  by  a  PQT.  In  the  authors1  words: 

"As  each  computer  program  component  is  added  and  each  PQT  conducted,  increased 
confidence  develops  in  the  CPCEI  being  tested." 

Such  sources  provide  a  basis  for  regarding  PQTs  as  having  primarily  a  "con¬ 
fidence-building,"  rather  than  qualification,  purpose.  It  is  suggested  that 
the  concept  might  be  further  amplified  along  the  following  lines: 

a.  PQTs  should  be  planned  as  formal  events  which  will 
enable  the  SPO  to  verify  contractor  progress  in 
developing  a  computer  program  CEI,  prior  to  its 
formal  qualification. 

b.  A  PQT  should  be  regarded  as  a  demonstration,  rather 
than  a  true  "test,"  since  it  is  presumed  that  the 
contractor  will  have  accomplished  the  necessary 
assembly  testing  for  the  components  in  question  as 
part  of  his  CPT&E  effort.  However,  sessions  could 
be  arranged  to  provide  SPO  observers  the  opportunity 
to  verify  additional  detailed  requirements  where 
indicated,  perhaps  on  a  sampling  basis. 

c.  Scheduling  should  be  based  on  the  contractor’s 
CPCEI  development  plan,  such  that  PQTs  are  timed 
to  coincide  with  stages  at  which  key  components, 

or  assemblies  of  components,  are  due  to  be  completed 

i 

*  Piligian,  M.  S.  and  Pokorney,  J.  L. ,  Air  Force  Concepts  for  the  Technical 
Control  and  Design  Verification  of  Computer  Programs.  Electronic  Systems 
Division  Technical  Report,  ESD-TR-67-67 ,  April  1967. 
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at  an  operable  level.  Scheduling  should  be  spaced, 
to  provide  visibility  of  progress  at  early,  inter¬ 
mediate,  and  later  points  during  Acquisition. 

d.  There  should  be  no  necessary  requirement  to  subject 
all  components  of  a  complex  CPCEI  to  PQTs.  Emphasis 
should  be  placed  on  components  and  assemblies  which 
are  critical  and/or  which  will  provide  adequate  in¬ 
dices  of  contractor  progress  in  meeting  the  CPCEI 
schedule  and  design  objectives.  Considering  time 
and  cost,  it  is  suggested  that  3  or  4  judiciously 
spaced  PQTs  would  have  accomplished  the  purpose  for 
the  BUIC  III  ADP,  instead  of  the  35  which  were  actually 
held. 

3.  As  noted  above,  the  emphasis  on  PQTs  in  BUIC  III  resulted  from  an 
initial  recognition  that  the  FQT,  which  was  planned  (for  ADP)  to  be  held  during 
a  short  period  between  the  installation/checkout  of  computer  programs  at  the 
Category  II  site  and  the  beginning  of  Category  II  tests,  would  not  be  adequate 
to  accomplish  the  full  qualification  objectives.  The  question  of  how  those 
objectives  can  be  met,  other  than  by  shifting  the  burden  to  PQTs,  does  not  have 
answers  which  are  altogether  obvious.  Consideration  has  to  be  given  to  the  wide 
variations  among  CPCEIs  which  can  exist  with  respect  to  such  characteristics 
as  size  and  complexity,  stability  of  configuration  once  developed  and  criticality 
of  performance  characteristics  during  operational  employment. 

Generalized  requirements  should  therefore  permit  appropriate  solutions  to  be 
formulated  on  a  case-by-case  basis.  However,  for  computer  programs  like  ADP 
and  SEP,  it  is  believed  that  better  solutions  could  have  been  reached  within 
the  framework  of  provisions  which  now  exist,  e.g.,  in  ESD  Exhibit  EST-1  and 
elsewhere.  It  should  be  recognized  that  the  initial  burden  of  arriving  at 
feasible  and  adequate  solutions  for  a  given  CPCEI  falls  on  the  contractor,  who 
must  propose  the  solutions  in  the  process  of  formulating  test  requirements  to 
be  contained  in  Section  4  of  the  Part  I  specification.  Alternatives  which 
might  have  been  considered,  but  were  not  tried  in  BUIC  III,  include  the 
following: 

a.  The  contractor  is  responsible  for  comprehensive  testing 
against  each  and  every  performance  requirement  of  the 
computer  program,  as  part  of  CPT&E.  If  the  development/ 
testing  process  is  properly  managed,  it  should  be  possible 
to  utilize  CPT&E  as  a  source  of  data  for  qualifying  much 
of  the  detail,  in  particular,  which  can  be  time-consuming 
and  expensive  to  repeat  during  formal  test/demonstrations. 

To  exploit  this  possibility,  the  contractor  should  be 
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required  to  demonstrate  the  effectiveness  of  internal 
management  controls,  as  well  as  to  furnish  data  and/or 
other  appropriate  evidence  that  individual  requirements 
are  satisfied.  Direct  verification  by  the  SPO  of  de¬ 
tailed  requirements  in  selected  areas  could  be  carried 
out  on  a  limited  sampling  basis  during  PQTs  and  FQT. 

b.  An  FQT  which  is  held  at  the  Category  II  site  is  necessarily 
limited.  Appropriately,  as  in  the  case  of  .ADP,  it  should 
emphasize  testing  of  major  requirements  for  the  integrated 
CPCEI  performing  in  operationally  configured  equipment  with 
live  inputs.  However,  there  is  reason  to  believe  that  it 
should  have  been  feasible  to  accomplish  some  of  the  ob¬ 
jectives  for  ADP  earlier.  For  example,  much  of  the  inter¬ 
face  testing  with  the  maintenance-diagnostic  computer  pro¬ 
gram,  and  the  simulation  mode  test,  would  have  been  possible 
to  accomplish  in-plant,  with  proper  advance  planning.  Such 
objectives  as  Live  Mode  and  System  Load  definitely  required 
the  Category  II  facilities.  However,  to  the  degree  that 
valid  tests  are  made  possible  by  available  simulation,  equip¬ 
ment,  and  other  relevant  factors,  they  should  be  accomp¬ 
lished  by  FQTs  held  in-plant. 


F.  BUIC  III  CYCLE  TIME  PROBLEM 

1 .  Statement  of  the  Problem 

The  BUIC  III  "cycle  time  problem"  is  most  simply  described  as  excessive  oper¬ 
ating  time  of  the  BUIC  III  Air  Defense  Computer  Program  (ADP) .  During  ADP 
program  development,  analysis  and  testing  revealed  two  significant  aspects  of 
the  problem:  (1)  radar  inputs  processing;  and  (2)  operating  time  of  the  com¬ 
puter  program  components  that  perform  control,  guidance,  and  display  makeup. 

The  System  Specification,  SS-ES  416M  65-1B,  describes  timing  by  alluding  to 
it  in  the  performance  allocation  requirements  for  radar  inputs.  In  this  con¬ 
text,  a  correlation  period  is  specified  in  seconds;  two  correlation  periods 
equal  one  radar  scan.  Operating  time  was  specified  in  the  Part  I  Specification, 
CGTM2385A,  Volume  1,  as  seconds  minimum  to  seconds  maximum  per  cycle,  and 
seconds  minimum  to  seconds  maximum  per  bicycle.  The  minimum  value  was  an 
estimate  based  on  the  subframe  time  of  SAGE;  the  maximum  was  an  arbitrary 
value.  It  was  assumed  that  if  the  maximum  value  were  exceeded,  an  equipment  or 
program  malfunction  had  occurred  such  that  the  BUIC  Confidence  Diagnostic 
Program  (BCDP)  should  operate.  These  values  had  been  previously  established 
for  BUIC  II  and  were  then  applied  to  BUIC  III. 
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Estimated  cycle  time  of  the  BUIC  III  operational  program  was  initially  derived 
by  adding  increments  for  the  differences  between  BUIC  II  and  BUIC  III  features 
to  the  cycle  time  of  BUIC  II  which  had  been  determined  in  a  timely  study  em¬ 
ploying  reasonably  high  load  conditions.  The  differences  were  identified 
principally  in  input/output  and  computer  program  size. 

This  information  was  documented  in  TM-2385/000/00,  Preliminary  Design  of  BUIC 
III  Air  Defense  Program,  and  submitted  at  the  Preliminary  Design  Review  (PDR) 
held  January  1966.  At  the  time  of  the  PDR,  no  difficulty  was  anticipated  in 
meeting  the  cycle  time  constraints;  however,  input/output  table  size  had  been 
predicated  on  minimum  cycle  time.  This  table  design  made  inefficient  utiliza¬ 
tion  of  output  lines  with  the  possibility  of  system  degradation  due  to  loss  or 
delay  of  information  to  be  transmitted  to  interceptors  and  other  facilities. 

It  was  determined  that  input/output  table  size  should  be  based  on  projected 
cycle  time.  Accordingly,  SDC  was  committed  to  a  study  of  cycle  time  pro¬ 
jections  considering  I/O  time,  compression  of  input  data  to  one  word  format, 
and  the  effect  on  I/O  table  structure. 

In  the  months  that  followed  the  PDR,  SDC  did  perform  a  study  which  led  to  a 
new  prediction  of  cycle  time.  The  study  was  performed  using  a  load  based  on 
a  moderate  threat  situation  which  entailed  a  given  number  of  radar  returns/ 
cycle,  interceptor  capacity,  and  20  less  than  full  track  capacity.  The  re¬ 
sultant  prediction  as  of  the  end  of  May  1966  was  substantially  greater  than 
the  earlier  estimate  submitted  at  the  PDR.  At  this  point,  it  was  clear  that 
the  possibility  of  adverse  effects  on  system  performances  was  sufficient  to 
justify  continued  scrutiny  and  evaluation. 

Radar  Inputs  Processing 

The  continuing  study  ascertained  that  the  operation  of  the  radar  inputs  pro¬ 
cessing  program  (RIP)  consumed  a  relatively  large  amount  of  time.  RIP's 
operating  time  is  a  function  of  the  number  of  radar  returns  processed,  number 
of  returns  correlated,  and  other  factors.  As  a  necessary  part  of  this  study, 
the  possible  effects  of  high  cycle  time  were  determined  and  are  briefly 
summarized  here: 

a.  The  operation  of  ADP  would  be  interrupted  in  each  cycle 
exceeding  a  specified  threshold  number  of  seconds  since 
ADP  would  assume  a  malfunction  and  initiate,  BCDP  to 
investigate . 

b.  Degradation  of  active  and  passive  tracking  functions  could 
occur  since  tracking  parameters  had  been  optimized  based 
on  specified  minimum  cycle  time. 
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c.  Height  reply  rate  could  be  less  than  required  to  delay 
in  sending  out  height  requirements, 

d.  Response  to  operator  switch  actions  could  be  delayed. 

e.  Duration  of  forced  displays  would  vary,  thus  possibly 
causing  operator  confusion. 

f.  Guidance  calculations  for  interception  could  be  in  error. 

Concurrently  with  the  SDC  study,  MITRE  engaged  in  a  parallel  study  which  also 
identified  radar  return  processing  as  taking  a  large  amount  of  operating  time. 
Since  both  SDC  and  MITRE  agreed,  further  study  and  analysis  concentrated  upon 
radar  inputs  processing.  A  solution  was  recommended  to  the  SPO  and  approved 
that  entailed  a  redesign  of  the  radar  inputs  computer  program  components  such 
that  the  average  processing  time  for  each  radar  return  was  reduced;  additionally, 
a  new  program  system  design  was  developed  that  provided  for  operating  some 
functions  less  frequently  than  every  cycle  and  also  presented  a  new  data 
limiting  scheme.  This  change  was  implemented  as  ECP  54-1  and  was  included  in 
the  computer  program  when  Program  System  Test  (PST)  began. 

Control 


PST  of  ADP  was  performed  at  the  BUIC  Evaluation  Facility,  Hanscom  Field, 
Massachusetts,  using  live  inputs  to  the  computer  program.  The  testing 
revealed  that  the  cycle  time  problem  had  not  been  fully  resolved;  the  earlier 
prediction  of  operating  time  for  non-radar  processing  programs  was  not  valid. 
It  was  determined  that  the  underestimation  was  principally  in  the  amount  of 
time  the  control  program  (COP)  and  the  CPCs  performing  guidance  and  display 
makeup  took  to  operate.* 

BUIC  II  hardware  had  an  output  interrupt  every  70  ms.  that  allowed  the  control 
program  to  transfer  output  messages  as  well  as  read  the  message  processor. 

BUIC  III  hardware  has  an  additional  interrupt  every  30  ms.  for  processing  in¬ 
puts  from  the  message  processor.  The  high  operating  time  of  COP  was  a  result 
of  the  high  frequency  of  interrupts  being  processed  by  inefficient  interrupt 
processing  logic.  The  logic  of  COP  included  use  of  an  executive  loop  which, 
when  entered,  would  check  indicators  and  items  to  determine  the  need  to 
service  any  terminal  device,  the  need  to  start  any  I/O  operations,  the  need 
for  action  due  to  completion  of  an  I/O  operation  or  the  need  to  process  a 


*  Since  this  problem  is  treated  fully  in  TM-4153/001/00,  Final  Report  of  BUIC 
III  Timing  Analysis,  12/15/68,  no  attempt  is  made  here  to  address  the  problem 
in  its  entirety.  The  discussion  does  address  COP  because  of  its  unique 
aspects  and  significance. 
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clock  interrupt  or  an  external  request  interrupt.  When  any  interrupt  except 
an  equipment  failure  interrupt  occurred,  the  indication  of  the  interrupt  was 
set  and  the  executive  loop  was  entered.  If,  as  in  the  case  of  message  pro¬ 
cessor  service,  multiple  I/O  was  required,  several  passes  through  the  loop 
would  be  performed.  Because  of  the  large  amount  of  code  operated  in  this 
loop  and  the  frequency  of  messages  processor  interrupts  (one  every  21  ms.  in 
COP),  the  control  program  required  about  15/21  of  ADP  operating  time  for 
message  processor  servicing. 

This  problem  was  resolved  by  an  extensive  redesign  of  COP  to  make  it  more 
efficient  and  by  changes  to  the  message  processor  servicing  interval  (ECP  108) 
to  reduce  the  frequency  of  interrupts.  To  reduce  the  amount  of  time  required 
for  interrupt  processing  the  following  changes  were  made  in  the  control  program 
design:  Interrupts  as  much  as  possible  are  no  longer  queued,  but  are  com¬ 

pletely  processed  when  they  occur.  This  reduces  the  amount  of  code  operated 
when  an  interrupt  is  processed.  Secondly,  the  number  of  interrupts  per  unit 
time  is  reduced.  The  program  now  services  a  clock  interrupt  at  a  fixed  fre¬ 
quency  of  once  every  30  milliseconds  with  message  processor  servicing  per¬ 
formed  on  every  other  clock  interrupt.  The  routines  with  high  frequency  of 
operation  were  designed  and  coded  with  time  efficiency  the  main  concern.  This 
ECP  was  included  in  the  Timing  Version  subjected  to  Formal  Qualification 
Testing  at  the  BEF  2  December  -  13  December.  The  FQT  indicated  that  the 
Timing  Version  conforms  to  specification. 

2.  Conclusion 

The  cycle  time  problem  insofar  as  radar  inputs  processing  was  concerned  was 
identified  early  in  the  development  of  ADP  and  resolved  within  a  reasonable 
time;  of  greater  concern  is  the  late  realization  that  a  cycle  time  problem 
still  existed  that  had  its  origin  in  the  control  program  (COP) .  That  the 
latter  should  remain  undetected  until  PST  must  be  regarded  as  a  deficiency 
in  the  activities  preceding  PST.  Testing  during  CPT&E  of  COP  did  not 
sufficiently  approximate  its  exercise  in  a  live  environment,  the  difference 
being  principally  in  the  lack  of  live  inputs  that  would  actuate  the  processing 
routines  of  COP.  Hindsight  indicates  that  CPT&E  should  include  a  full 
exercise  with  attention  to  the  timing  of  the  control  program,  as  well  as  the 
other  CPCs  comprising  the  CPCEI,  in  a  manner  equivalent  to  employment  live. 

This  means  that  special  purpose  program  tools  must  be  designed  and  developed 
to  facilitate  such  testing.  Timing  tools  that  were  used  in  the  control  pro¬ 
gram  study,  redesign,  and  testing  are  described  in  Table  6-5. 

Experience  has  shown  that  two  critical  factors  must  be  considered  in  program 
design:  timing  and  storage.  Furthermore,  each  must  be  considered  in  relation 
to  the  other.  Although  operate  time  is  basically  a  direct  function  of  program 
size,  an  inverse  relation  may  also  exist.  For  example,  if  operate  time  had 
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Table  6-5.  Timing  Tools  Description 


1.  Ratio  of  COP  to  NON  COP  Time.  (COP  +  non  COP  =  total  ADP) . 

This  tool  was  developed  to  monitor  the  item  N005  in  COP. 

This  item  is  greater  than  0  whenever  COP  is  operating.  The 
software  routine  that  was  utilized  operated  in  the  non-ADP 
computer  and  generated  counts  of  when  ADP  was  in  COP  and 
out  of  COP. 

2 .  Ratio  of  Message  Processor  Time  Consumption  to  Total  ADP . 

Real  time  Clock  interrupts  were  blocked  and  a  program  loop 
was  placed  in  the  Manual  Inputs  Program  (MIN) .  The  item 
N005  in  COP  was  again  monitored  for  a  value  greater  than 
7  to  determine  the  percentage  of  time  required  for  message 
processor  servicing.  The  software  routine  that  was  developed 
operated  in  the  non-ADP  computer. 

3.  Interrupt  Count  and  Frequency  Recorder.  Two  tools  were 
developed  to  determine  the  frequency  and  interrupt  mix  in 
ADP.  Both  tools  utilized  were  developed  by  octally  modifying 
the  control  program  to  generate  counts  per  cycle  or  to  save 
the  index  register  setting  (XIR) .  The  XIR  was  saved  for  an 
interrupt  mix  and  sequence  analysis. 

4.  Increases  in  Cyclic  ADP  Due  to  Interrupts.  A  timed  loop  was 
placed  In  the  Telling  Program  (BIT) .  As  additional  interrupts 
were  allowed  to  occur  the  operating  time  of  BIT  was  recorded 
by  a  TIMER  program  In  the  non-ADP  computer.  The  difference 
between  the  expected  operating  time  of  BIT  (that  of  the  timed 
loop)  and  that  actually  recorded  indicated  the  time  increases 
due  to  interrupt  processing. 

5.  (Word  counts  of)  Message  Processor  Input  Operations.  A  routine 
was  generated  that  recorded  the  result  descriptor  of  message 
processor  inputs.  The  differences  between  the  maximum  message 
processor  word  count  and  that  of  the  actual  result  descriptor 
defined  how  many  words  were  actually  transferred. 
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no  ceiling,  a  computer  program  could  be  devised  that  uses  little  storage  but 
has  an  extremely  high  operate  time;  conversely,  if  storage  had  no  ceiling,  a 
computer  program  could  be  devised  that  uses  little  operate  time  but  requires 
large  storage.  Realistically,  of  course,  a  constraint  must  exist  for  both 
timing  and  storage.  An  optimum  balance  of  the  two  must  be  determined  con¬ 
sistent  with  other  system  constraints. 

The  fact  that  the  full  scope  of  the  cycle  time  problem  was  not  evident  until 
PST  can  be  attributed  in  part  to  the  emphasis  placed  on  meeting  the  computer 
program  storage  constraint.  Difficulty  was  experienced  in  confining  ADP  to 
the  available  storage.  Although  computer  program  storage  allocation  was  care¬ 
fully  reviewed  at  the  Critical  Design  Reviews  held  for  the  CPCs,  little 
attention  or  concern  was  expressed  with  regard  to  operate  time.  Some  questions 
were  asked  but  they  were  not  meaningfully  pursued.  Another  factor  that  may 
have  tempered  the  frame  of  mind  of  those  engaged  in  the  development  of  ADP  is 
the  previous  experience  of  SAGE.  High  operate  times  were  known  to  occur,  but 
little  had  been  done  to  ascertain  what  adverse  consequences  had  resulted.  A 
sort  of  mental  euphoria  had  set  in  such  that  the  potential  of  a  further 
timing  problem  was  assumed  to  be  inconsequential 

Frame  time  in  SAGE  is  the  equivalent  to  bicycle  time  in  BUIC  III.  Under  normal 
conditions,  SAGE  has  a  frame  time  of  15.7  seconds  but  instances  of  frame  time 
in  the  range  of  20  to  30  seconds  have  been  experienced  without  any  apparent 
effects,  and  are  not  uncommon.  Concern  has  been  expressed  by  personnel  newly 
assigned  to  SAGE  with  regard  to  high  frame  time;  however,  experienced  personnel 
have  learned  to  cope  with  it  (e.g.,  reduce  amount  of  radar  data  processed  by 
manual  switch  intervention).  Moreover,  much  of  the  high  frame  time  in  SAGE 
has  been  experienced  while  the  simulation  and  recording  functions  were 
operating,  both  of  which  increase  frame  time.  Neither  of  these  functions  would 
be  operated  during  a  hostile  attack.  This  analogy  between  SAGE  frame  time  and 
BUIC  bicycle  time  serves  to  illustrate  that  the  relation  of  operate  time  to 
system  performance  may  be  complex  and  equivocal. 

3.  Recommendations 


When  discussing  how  this  type  of  problem  may  be  prevented  or  at  least  detected 
earlier  in  the  future,  the  following  should  be  considered: 

a.  System  engineering  analyses  and  studies  preceding  program 
development  should  carefully  consider  system  requirements 
in  terms  of  response  time  desired  for  specified  actions 
and  situations,  the  frequency  of  functions,  required  data 
retention  and  allowable  data  loss,  and  similar  requirements 
that  influence  computer  program  operate  time.  However,  the 
emphasis  should  be  on  the  basic  requirements,  not  on  operate 
time  per  se  which  is  derived  in  satisfying  the  requirements. 
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The  studies  should  consider  alternatives  and  determine 
potential  consequences  of  less  than  optimum  performance. 

b.  Specification  of  computer  program  requirements  should 
include  a  description  of  specific  load  conditions  that 
influence  operate  time  for  each  computer  program  function. 
Actual  values  should  be  specified  and  related  explicitly 
to  the  expected  response  of  the  CPCEI. 

c.  The  proposed  program  design  presented  at  the  time  of 
Preliminary  Design  Review  should  include  a  description 
of  the  expected  operate  time  of  each  CPC  and  the  CPCEI 
as  a  whole  under  the  varying  load  conditions  specified. 

d.  Personnel  engaged  in  the  management  review  process 
should  examine  and  evaluate  information  based  on  test 
data  regarding  operate  time  of  the  CPCs  during  program 
development.  Preliminary  Qualification  Tests  can  serve 
as  the  vehicle  for  this  purpose. 

e.  Provision  should  be  made  for  the  development  and  use  of 
computer  program  test  tools  that  clock  the  operate  time 
of  sets  of  code.  Other  special  purpose  timing  tools  may 
also  be  desired. 

f.  Following  the  initial  testing  of  the  CPCs  (parameter  and 
assembly)  typically  performed  by  the  programmers  who  did 
the  original  coding,  testing  should  be  performed  by  an 
independent  group  of  programmers  to  offset  the  natural 
bias  of  the  original  programmers. 

g.  It  is  evident  that  some  problems  by  their  nature  are  not 
readily  detected  in  a  simulated  environment;  therefore, 
it  is  desirable  to  exercise  the  CPCEI  in  a  live  environ¬ 
ment  at  the  earliest  reasonable  time. 


101 

(Page  102  blank) 


CHAPTER  VII 


SUMMARY  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

Systems  management  techniques  based  on  375-series  principles  were  adapted  and 
applied  comprehensively  for  the  first  time,  in  BUIC  III,  to  the  acquisition 
of  system  information  processing  elements.  The  first  part  of  this  report 
is  devoted  to  a  review  of  the  background  of  those  techniques  in  terms  of  Air 
Force  systems  management  concepts  and  trends,  and  relating  them  to  earlier 
practice  at  ESD  in  the  system  programs  which  preceded  BUIC  III.  A  second  part 
of  the  report  is  devoted  to  describing  the  elements  of  contractor  effort, 
identifying  the  novel  management  requirements,  and  summarizing  the  milestones 
associated  with  the  requirements  as  they  actually  occurred  during  the  course 
of  the  BUIC  III  acquisition.  Finally,  discussions  are  presented  of  the  con¬ 
tractor  successes  and  difficulties  in  implementing  the  requirements,  in  the 
attempt  to  identify  and  record  aspects  of  the  BUIC  III  experience  that  can  be 
usefully  related  to  future  system  acquisitions. 

This  chapter  briefly  summarizes  selected  highlights  of  the  findings,  primarily 
in  terms  of  comments  and  recommendations  concerning  the  formally  documented 
management  techniques.  Emphasis  is  placed  on  the  formal  requirements,  not  only 
because  they  were  indeed  the  focal  topic  of  the  study,  but  because  they  provide 
the  most  direct  vehicle  by  which  results  of  experience  can  be  brought  to  bear 
on  future  practice.  At  the  time  this  report  is  being  written,  some  of  the 
contractual  exhibits  and  supporting  data  items  are  in  process  of  being  revised 
to  conform  with  recently  promulgated  Defense  Department  standards  for  configura¬ 
tion  management.  Similar  revisions  will  have  to  be  made  in  the  related  375- 
series  manuals  and  other  documents.  Based  on  current  information,  it  appears 
that  many  of  the  familiar  Air  Force  terms  —  e.g.,  names  of  baselines,  specifi¬ 
cations,  inspections  —  will  be  replaced  by  the  standard  DOD  terminology,  but 
that  the  existing  Air  Force  practices  may  otherwise  be  retained  essentially 
intact.  Hence,  it  is  assumed  that  lessons  learned  from  the  application  of 
systems  management  techniques  as  discussed  herein  will  continue  to  be  pertinent 
following  translation  into  the  new  framework  of  manuals  and  exhibits. 


B.  CONFIGURATION  MANAGEMENT 

Configuration  management  requirements  in  BUIC  III  were  contractually  specified 
by  a  special  BUIC  III  exhibit  which  was  equivalent  in  all  essential  respects 
to  ESD  Exhibit  EST-1  and  applicable  portions  of  AFSCM  375-1.  Basic  procedures 
relating  to  control  of  changes,  specification  maintenance,  and  accounting  were 
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similar  to  those  which  had  been  developed  and  used  prior  to  BUIC  III  except  for 
differences  in  standard  forms  and  the  fact  that  control  had  not  previously  been 
applied  at  the  product  configuration  baseline  level.  In  general  —  i.e.,  as 
regards  configuration  management  aspects  other  than  the  content  of  specifi¬ 
cations  —  the  procedures  were  applied  to  effectively  control  and  process  a 
large  volume  of  changes  during  the  course  of  the  program,  with  relatively  minor 
difficulties.  Recommendations  below  are  based  on  applications  of  EST-1  to 
other  systems  as  well  as  on  the  BUIC  III  experience. 

1.  Having  all  configuration  management  requirements  which  apply  to  com¬ 
puter  programs  in  a  self-contained  document,  as  was  true  in  BUIC  III,  is 
desirable.  Although  it  is  presently  a  separate  document  in  its  interim  form  as 
an  ESD  Exhibit,  EST-1  is  formated  as  change  pages  to  AFSCM  375-1  and  is  more 
difficult  for  people  concerned  solely  with  CPCEIs  to  use.  If  and  when  EST-1  and 
375-1  (or  their  successors)  are  combined  into  a  single  document  covering  all 
types  of  system  items,  the  difficulty  may  be  increased,  particularly  for  those 
contractors  who  do  not  have  a  backlog  of  experience  in  distinguishing  the 
applicable  principles  and  practice  as  between  computer  program  and  equipment 
items.  A  separate  and  self-contained  manual  for  computer  programs,  containing 
explanations  and  examples  in  addition  to  the  bare  definitions  of  legal  require¬ 
ments,  would  facilitate  contractual  application  and  use  by  both  SPO  and  con¬ 
tractor  personnel. 

2.  The  use  of  a  common,  standard  format  for  certain  forms  —  e.g., 
Engineering  Change  Proposal  —  is  now  required  for  both  equipment  and  computer 
program  items.  There  is  no  advantage  to  this  particular  standardization;  in 
fact,  it  encourages  expectations  of  similarities  which  do  not  exist.  It  is 
recommended  that  separate  forms  appropriate  to  the  purpose  be  devised  and  used, 
in  the  interests  of  improving  efficiency  .and  avoiding  misinterpretations. 


C.  SPECIFICATIONS 

Requirements  for  the  two-part  CPCEI  specification  imposed  in  BUIC  III  were 
essentially  those  which  are  currently  described  in  EST-1  and  associated  Forms 
9  (now  DD  1664s),  and  for  which  supplementary  experience  is  also  available  from 
their  use  in  other  programs.  Based  on  these  experiences  certain  recommended 
lines  of  improvement  are  suggested  in  Chapter  VI  preceding,  and  below.  However, 
to  maintain  perspective,  the  problems  identified  should  be  construed  as 
indicating  only  that  there  are  specific  areas  needing  further  study  and  refine¬ 
ment,  not  that  basic  changes  are  being  proposed.  It  is  believed  that  the  two- 
part  specification  has  been  demonstrated  to  be  the  key  element  in  developing 
an  effective  framework  for  managing  computer  program  acquisition,  and  that  the 
existing  structure  of  requirements  is  superior  to  various  proposed  alternatives. 

The  types  and  levels  of  information  to  be  supplied  in  the  Part  I  specification 
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are  generally  appropriate  and  necessary  in  order  for  the  Part  I  to  accomplish 
its  various  functions,  namely:  of  governing  the  computer  program  design, 
development,  and  testing;  controlling  interfaces  of  the  CPCEI  with  other  items; 
and  serving  as  the  primary  baseline  for  CPCEI  configuration  management.  The 
Part  II  meets  the  important  needs  of  providing  comprehensive  documentation 
which  can  be  used  by  computer  programmers  for  diagnosing  malfunctions  and 
designing  changes,  as  well  as  providing  a  precise  technical  definition  of  the 
CPCEI  for  configuration  management  at  the  product  baseline  level.  However, 
problems  needing  further  study  and  clarification  include: 

1.  The  degree  to  which  considerations  of  computer  program  design  are  in¬ 
corporated  into  the  content  and  structure  of  the  Part  I  specification  should 
be  determined.  Questions  exist,  on  the  one  hand,  whether  mathematical  for¬ 
mulas  and  equations  representing  design  solutions,  which  may  conflict  with 
requirements  expressed  purely  in  performance  terms,  should  be  called  for  in 
the  Part  I  except  where  their  firm  intent  is  to  establish  design  constraints. 

On  the  other  hand,  the  question  has  also  been  raised  whether  the  overall  com¬ 
puter  program  design  structure  should  not  be  anticipated,  verified,  and  est¬ 
ablished  in  a  corresponding  structure  of  Part  I  functional  elements,  to  provide 
an  improved  basis  both  for  managing  the  computer  program  development  and  for 
testing. 

2.  The  scope  of  intended  coverage  under  Data  Base  Requirements  (3.1.3) 
in  the  Part  I  specification  should  be  determined,  and  a  number  of  terms  should 
be  defined.  Questions  needing  resolution  relate  to  data  which  are  input  prior 
to  computer  program  operation  vs.  variable  values  input  during  the  course  of 
operation,  computational  constants  vs.  data  to  be  processed,  input/output  vs. 
intermediate  data,  and  others.  The  term  "data  base'1  is  subject  to  a  variety 
of  interpretations  with  respect  to  these  considerations.  Also,  a  number  of 
terms,  e.g.,  ’'adaptation, 11  "parameter,"  etc.,  need  to  be  defined  and  supported 
by  explanations  of  their  significance  in  terms  of  functions  served  by  the  speci¬ 
fication  . 


3.  The  level  of  descriptive  detail  contained  in  the  Part  II  specifications 
should  be  examined.  Experience  has  indicated  that  the  Form  9  (DD  1664)  for 

the  Part  II  specification  is  comprehensive  and  technically  sound.  However,  as 
is  true  of  Forms  9  in  general,  its  application  should  be  carefully  tailored 
to  the  given  CPCEI,  taking  into  account  the  intended  uses  and  maintenance  of 
the  document  following  its  acceptance.  The  value  of  detailed  prose  and  flow 
chart  materials  at  the  CPC  level,  in  particular,  which  are  expensive  to  pro¬ 
duce  and  maintain,  should  be  examined  critically,  and  the  required  detail 
reduced  to  a  minimum  level  which  is  consistent  with  the  needs  of  the  given 
application. 

4.  Implied  functions  of  the  Part  II  specification  as  a  contractual  require¬ 
ments  instrument  should  be  clarified.  The  Part  II  computer  program  specifica¬ 
tion  is  recognized  in  EST-1  to  be  an  "as-built"  technical  description  of  the 
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CPCEI  configuration.  However,  the  wording  and  content  of  certain  sections  in 
the  Form  9  are  not  fully  consistent  with  that  role,  e.g.,  as  contrasted  with 
functions  which  an  equipment  Part  II  specification  would  normally  serve  in 
governing  production,  acceptance,  and  delivery,  following  the  first  article. 
Recommended  changes  which  might  help  to  avoid  confusion  and  misinterpretations 
include:  under  Section  1,  "SCOPE" ,  eliminate  emphasis  on  units  subsequent  to 
the  FACI’d  article  and  on  CPCEI  serial  numbers;  as  a  title  for  Section  3, 
substitute  "TECHNICAL  DESCRIPTION"  for  "REQUIREMENTS";  eliminate  Sections  A  and 
5  completely,  as  types  of  requirements  which  should  be  contractually  specified 
independently  of  the  Part  II  CPCEI  specification,  prior  to  delivery  of  the 
first  article. 


D.  DESIGN  REVIEWS 

A  Preliminary  Design  Review  (PDR)  was  held  for  each  of  the  three  major  BUIC  III 
CPCEIs,  essentially  in  accordance  with  the  guidelines  set  forth  in  ESD  Exhibits 
EST-1  and  EST-3.  These  were  conducted  successfully  and  were  regarded  as  being 
useful  and  productive  sessions  by  representatives  of  both  the  SPO  and  contractor. 

Critical  Design  Reviews  (CDRs)  were  conducted  incrementally  in  combination  with 
Preliminary  Qualification  Tests,  at  the  level  of  completed  design  for  the  com¬ 
ponents  undergoing  review.  These  provided  opportunities  for  review  and  comment 
by  representatives  of  the  SPO,  and  were  considered  useful.  Although  some  of 
the  discussions  are  reported  to  have  resulted  in  minor  changes,  the  CDRs  were 
not  associated  with  formal  actions. 

As  a  general  matter,  questions  continue  to  be  raised  regarding  objectives,  in 
terms  of  formal  actions,  which  the  computer  program  CDR  is  intended  to  accomplish. 
Established  objectives  for  equipment  —  e.g.,  release  of  design  fo'r  production 
and  as  a  basis  for  support  items  —  do  not  apply.  Available  guidance  in  EST-1 
and  EST-3  indicates  that  the  only  analogous  purpose  for  CPCEIs  is  to  identify 
documentation  which  will  be  "released  for  coding  and  testing",  a  purpose  which 
can  only  exist  when  CDRs  are  held  at  the  flow  chart  level.  At  that  stage  the 
development  is  only  partially  complete  and  no  testing  can  have  been  accomplished; 
implications  relating  to  formal  approval/disapproval,  and  applicability  of 
configuration  controls,  have  become  matters  of  debate.  The  problem  deserves 
further  attention.  In  the  interim,  it  is  suggested  that  in  the  case  of  com¬ 
puter  programs,  CDRs  should  be  regarded  frankly  as  monitoring  events  only,  at 
a  level  between  PDR  and  FACI,  having  the  primary  purpose  of  providing  visi¬ 
bility  of  the  development.  A  change  in  the  name  would  be  a  helpful  step  towards 
alleviating  the  prevalent  confusion  with  accepted  practices  associated  with 
CDRs  for  equipment  items. 
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E. 


CATEGORY  I  TESTING 


Requirements  for  Category  I  testing  of  CPCEIs  encompassed:  identification  of 
test  requirements  in  the  Part  I  CPCEI  specifications;  preparation  of  Category 
I  Test  Plans;  preparation  of  PQT/FQT  Test  Procedures  and  Reports;  and  conduct 
of  the  formal  test  sessions.  As  a  whole,  these  represented  elements  of  con¬ 
tractor  effort  which  were  novel  in  BUIC  III  as  compared  with  practice  in 
earlier  systems.  Difficulties  were  encountered  in  carrying  out  the  test  pro¬ 
gram  which  are  attributed  to  absence  of  previous  contractor  experience  with 
the  concepts  and  procedures,  as  well  as  to  certain  ambiguities  in  the  require¬ 
ments  as  they  were  formulated  at  the  time  of  contractual  application. 

The  principal  difficulties  were  associated  with  PQTs ,  which  were  carried  out  in 
a  total  of  57  testing  sessions  for  two  of  the  CPCEIs  (ADP  and  SEP)  and  were 
designed  to  provide  comprehensive  verification  of  performance  relating  to 
detailed  characteristics  specified  in  the  Part  I  specifications.  The  PQTs 
proved  to  be  costly  in  time  and  effort  to  conduct  and  were  not  particularly 
satisfactory  in  meeting  their  objectives. 

In  the  case  of  FQTs,  the  principal  objective  was  to  demonstrate  compliance 
with  major  performance  requirements  of  the  computer  programs  during  integrated 
system  operation.  Although  some  problems  were  encountered  with  documentation 
and  test  conduct,  particularly  during  early  attempts,  the  FQTs  were  compara¬ 
tively  successful.  In  contrast  with  the  PQTs,  they  were  readily  recognized  by 
the  contractor  personnel  involved  as  being  formal  tests  which  serve  useful 
and  necessary  purposes,  as  well  as  being  feasible  to  accomplish. 

Comments  and  recommendations  based  on  discussions  contained  in  the  preceding 
chapter  are  summarized  briefly  below. 

1.  A  number  of  factors  can  be  identified  as  contributing  to  the  severe 
problems  experienced  in  carrying  out  the  BUIC  III  PQTs.  In  some  part  these 
were  matters  of  contractor  management  and  technical  practices  which  had  not 
been  geared  to  meeting  the  complex  requirements  of  formal  Category  I  testing, 
and  which  did  not  receive  the  vigorous  attention  they  needed  until  late  in  the 
program.  However,  the  authors  believe  that  the  initial  approach  which  the  SPO 
and  contractor  devised  jointly  at  an  early  point  was  basically  unsound.  The 
PQTs  planned  were  far  too  many  and  attempted  to  be  too  comprehensive  in  their 
coverage;  they  should  have  been  regarded  and  planned  as  interim  demonstrations 
of  contractor  progress  towards  accomplishing  the  development,  not  as  means  of 
qualification . 

2.  To  be  realistic,  qualification  requirements  must  be  tailored  to 
individual  cases.  The  degree  to  which  comprehensive  formal  verification  is 
feasible,  and  justifies  its  cost,  should  be  recognized  as  a  variable  which 
depends  on  such  considerations  as  size  and  complexity  of  the  CPCEI,  frequency 
and  extent  of  changes,  and  criticality  of  perfect  performance  in  initial 
operational  use. 
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3.  The  contractor  should  be  required  to  demonstrate  internal  management 
procedures  which  insure  that  Part  I  specification  requirements,  including 
approved  changes,  are  incorporated  into  the  computer  program  design  and  tested 
routinely  during  CPT&E.  Where  conditions  permit,  documented  CPT&E  procedures 
and  results  should  be  utilized  as  a  source  of  data  for  qualifying  detailed 
performance/design  characteristics  of  the  computer  program  components. 

4.  Where  feasible,  it  is  desirable  to  conduct  FQTs  using  the  integrated 
CPCEI  in  the  system  environment.  To  the  degree  that  adequate  simulation  is 
available,  FQTs  or  partial  FQTs  should  be  performed  at  the  contractor's 
facility. 


F.  CONCLUSION 

As  a  major  summary  result,  this  study  has  indicated  that  the  principles  and 
content  of  the  management  requirements  are  not  only  sound  but  represent 
necessary  and  substantial  improvements  over  previous  practice.  Their  appli¬ 
cation  in  BUIC  III  occasioned  problems,  principally  in  the  areas  of  documenta¬ 
tion  and  testing,  which  are  discussed  at  length  in  the  preceding  chapter.  In 
general,  however,  the  problems  can  be  related  most  directly  to  the  fact  that 
the  procedures  were  novel,  rather  than  unsound,  and  the  BUIC  III  personnel 
were  inexperienced  with  their  use.  The  original  intent  was  to  initiate  the 
adoption  of  standards  which  would  enable  computer  programming  to  be  managed 
in  a  uniform  manner,  consistently  with  the  management  of  equipment  and  other 
elements  in  a  system  program.  The  matters  of  uniformity  and  continuity  in 
documentation  and  procedures  are  the  key  elements  which  make  it  possible  for 
both  SPO  and  contractor  to  benefit  from  experience,  from  system  to  system.  In 
the  authors1  opinion,  these  advantages  have  become  more  evident  as  the  pro¬ 
cedures  have  become  better  understood  and  established,  particularly  through 
continued  use  in  other  system  programs. 

By  the  same  token,  BUIC  III  clearly  demonstrated  that  the  initial  acquisition 
of  contractor  capability  to  implement  the  requirements  adequately  can  be 
difficult  and  time-consuming.  In  part,  some  of  the  problems  can  be  ascribed 
to  the  fact  that  the  total  375-series  framework  was  a  novelty  to  the  con¬ 
tractor  at  the  time  BUIC  III  began,  and  many  of  its  implications  for  internal 
management  were  not  appreciated  until  late  in  the  program.  There  is  evidence 
that  difficulties  could  have  been  avoided  by  better  initial  guidance,  e.g., 
in  interpreting  the  formal  test  requirements,  and  by  systematic  training  of 
personnel  in  relevant  documentation  and  procedures.  Misinterpretations  and 
absence  of  common  understanding  of  objectives  and  requirements  were  frequent, 
both  internally  among  contractor  personnel  and  in  relations  with  the  SPO. 

In  general,  as  is  indicated  in  comments  and  recommendations  made  above,  pro¬ 
blems  can  be  traced  to  a  variety  of  sources.  However,  it  is  believed  that 
training  of  personnel,  based  on  adequate  uniform  guidance,  will  continue  to 
be  a  fruitful  area  for  increased  future  attention. 
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APPENDIX  A 


GLOSSARY  OF  ABBREVIATIONS 


AC&W 

- 

Aircraft  Control  and  Warning 

ADC 

- 

Aerospace  Defense  Command  (formerly  Air  Defense  Command) 

ADP 

- 

Air  Defense  (Computer)  Program 

AFL 

- 

Air  Force  Letter 

AFLC 

- 

Air  Force  Logistics  Command 

AFR 

- 

Air  Force  Regulation 

AFS 

- 

Air  Force  Specialty 

AFSC 

- 

Air  Force  Specialty  Code 

AFSC 

- 

Air  Force  Systems  Command 

AFS  CM 

- 

Air  Force  Systems  Command  Manual 

AGE 

- 

Aerospace  Ground  Equipment 

AI&C 

- 

Adaptation,  Installation,  and  Checkout 

APASTO 

- 

Air  Defense  Computer  Program  and  System  Training  Office 

ARDC 

- 

Air  Research  and  Development  Command  (predecessor  of  AFSC) 

AS  PR 

- 

Armed  Services  Procurement  Regulation 

ATC 

- 

Air  Training  Command 

BARS 

- 

BUIC  Analysis  and  Reduction  System 

BCDP 

- 

Backup  Confidence  Diagnostic  Computer  Program 

BEF 

- 

BUIC  Evaluation  Facility 

BEPS 

- 

BUIC  Exercise  Preparation  System 

BMF 

- 

BUIC  Management  File 

BSAO 

- 

Burroughs  Site  Activation  Office 

BSD 

- 

Ballistic  Systems  Division 

BUDR 

- 

BUIC  Data  Reduction 

BUIC 

- 

Backup  Interceptor  Control 

CCB 

- 

Configuration  Control  Board 

CCBD 

- 

Configuration  Change  Board  Directive 

CCDSO 

- 

ADC  Command  and  Control  Defense  Systems  Office 

CDP 

- 

Contract  Definition  Phase 

CDR 

- 

Critical  Design  Review 

CDRL 

- 

Contract  Data  Requirements  List 

CEI 

- 

Contract  End  Item 

CMO 

- 

Configuration  Management  Office 

CC 

- 

Control  Center 

CP 

- 

Change  Proposal 

CPC 

- 

Computer  Program  Component 

CPCEI 

- 

Computer  Program  Contract  End  Item 

CPT&E 

- 

Computer  Program  Test  and  Evaluation 

CR 

- 

Change  Report  (In  BUIC  III,  reporting  a  Class  II  change) 

CR 

Change  Recommendation  (In  BUIC  II,  a  recommended  change  to  any 
system  element,  prepared  by  any  participating  agency) 

109 


DC  -  Direction  Center 


DOD 

Department  of  Defense 

ECP 

ERP 

ESD 

ESS 

FAC  I 

FQT 

FSPS 

Engineering  Change  Proposal 

Error  Recovery  Program 

Electronic  Systems  Division 

Experimental  SAGE  Sector 

First  Article  Configuration  Inspection 

Formal  Qualification  Test 

Field  Site  Production  System 

GEEIA 

GFP 

GSE/TDC  - 

Ground  Electronics  Engineering  and  Installation  Agency 
Government-Furnished  Property 

General  Systems  Engineering/Technical  Direction  Contractor 

HE 

Human  Engineering 

IAC 

Integrating  and  Assembly  Contractor 

LET 

LLO 

LS 

Live  Environment  Testing 

Lexington  Liaison  Office 

Life  Support 

MSTP 

Manual  System  Training  Program 

NCC 

NORAD 

NRD 

NORAD  Control  Center 

North  American  Air  Defense  Command 

National  Range  Division 

OR 

Operational  Requirement 

PAGE 

PCB 

PDR 

PECP 

PED 

PERT 

PQT 

PS 

PSTE 

PSPP 

PST 

PTDP 

Primary  Air  Ground  Environment 

Product  Configuration  Baseline 

Preliminary  Design  Review 

Preliminary  Engineering  Change  Proposal 

Personnel-Equipment  Data 

Program  Evaluation  and  Review  Technique 

Preliminary  Qualification  Test 

Personnel  Subsystem 

Personnel  Subsystem  Test  and  Evaluation 

Proposed  System  Package  Plan 

Program  System  Test 

Preliminary  Technical  Development  Plan 

QQPRI 

Qualitative  and  Quantitative  Personnel  Requirements  Information 
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RDT&E 

- 

Research,  Development,  Test  and  Evaluation 

RFP 

- 

Request  for  Proposal 

SAC 

- 

Strategic  Air  Command 

SAC 

- 

Site  Activation  Contractor 

SAGE 

- 

Semi-Automatic  Ground  Environment  (System) 

SAWG 

- 

System  Acquisition  Working  Group 

SCL 

- 

Specification  Change  Log 

SCN 

- 

Specification  Change  Notice 

SATAF 

- 

Site  Activation  Task  Force 

SDC 

- 

System  Development  Corporation 

SDR 

- 

System  Design  Review 

SEP 

- 

System  Exercise  (Computer)  Program 

SETE 

- 

System  Exercising  for  Training  and  Evaluation 

SOW 

- 

Statement  of  Work 

SPD 

- 

System  Program  Director 

SPO 

- 

System  Program  Office 

SPP 

- 

System  Package  Program 

SSTP 

— 

SAGE  System  Training  Program 

TAC 

— 

Tactical  Air  Command 

TC 

- 

Training  Concept 

TED 

- 

Training  Equipment  Development 

TEPI 

- 

Training  Equipment  Planning  Information 

TM 

- 

Technical  Memorandum 

TP 

- 

Training  Plan 

UCP 

- 

Utility  Computer  Program 

USAF 

- 

United  States  Air  Force 

USCN 

USP 

- 

Uniform  Specification  Change  Notice 

Uniform  Specification  Program 

VDE 

- 

Variable  Display  Equipment 

Ill 
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